Queues are commonly used to lineup work to be done. Consider a queue at a store. The first person in line is the first to be serviced…

This post explains explicit data definitions, how they are leveraged to make integration safe and easy. Finally the post contrasts explicit…

Flow and implmenetation methods are 2 types of functions which help create manintainable software. Flow methods contain a series of program…

Schema-on-read places the burden of imposing the structure and types of data on the consumer (reader) of data. Schema-on-read decouples a…

Data operations is the act of keeping data pipeline operational, and the data flowing. Data operational maturity focuses on answering: Is a…

Large data deployments often contain multiple levels of derived tables. Understanding lineage is critical for resolving data downtime and…

Snowflake does not currently support sub query pruning which can have serious cost implications for DBT incremental updates against…

Views are an important tool in legacy warehouse environments. Views act as an interface which decouples clients (like tableau, looker…

TL;DR I hit the 1024 File descriptor limit in lambda. AWS has a bug handling this case, in which END and REPORT logs were omitted. The…

Engineering accounting is a prerequisite to predictability in the same way standard accounting is a prerequisite to financial…

Freshness checks are an easy way to see if your data is out of date. A freshness check queries a table for the MAX created time, and…

Cleaning up resources is one fo the most difficult things in Go. If careful attention isn't paid to files, network connections, or…

This article lists a number snowflake configuration csv copy into parameters and the outputs they produce. I'm currently working on a…

The low cost of storage combined with systems like Presto, which decouple computer from storage, makes it easier than ever to get started…

When I started using Spectrum and Athena I was confused by partitions. I found few resources and examples on how partitions actually worked…

Last summer I grew sprouts. I've always enjoyed gardening but am currently living in an apartment with very little light exposure. Due to…

Probing is a technique to perform regular checks on a service using a short interval. Probes provide signals that can significantly cut down…

Identifying system boundaries is one of the hardest parts of understanding and modifying systems. A focus on goals acts as a compass by…

Systems exist all around us. Every day we operate in an incalculable number of systems. Our biological systems dictate our physical needs…

Self-service tools enable predictable delivery while still allowing for organizational oversight. Self-service tools allow customers to…