Schema On Read Considered Harmful
May 29, 2021Schema-on-read places the burden of imposing the structure and types of data on the consumer (reader) of data. Schema-on-read decouples a…
Data Operational Maturity
May 20, 2021Data operations is the act of keeping data pipeline operational, and the data flowing. Data operational maturity focuses on answering: Is a…
Deriving Data Lineage From SQL Statements
May 14, 2021Large data deployments often contain multiple levels of derived tables. Understanding lineage is critical for resolving data downtime and…
Beware of DBT Incremental Updates Against Snowflake External Tables
April 10, 2021Snowflake does not currently support sub query pruning which can have serious cost implications for DBT incremental updates against…
Using Views for Backwards Compatible Data Migrations
April 01, 2021Views are an important tool in legacy warehouse environments. Views act as an interface which decouples clients (like tableau, looker…
AWS Lambda - Debugging File Descriptor Exhaustion Using Go
March 10, 2021TL;DR I hit the 1024 File descriptor limit in lambda. AWS has a bug handling this case, in which END and REPORT logs were omitted. The…
Toward Predictable Engineering Velocity - Accounting
February 25, 2021Engineering accounting is a prerequisite to predictability in the same way standard accounting is a prerequisite to financial…
Stay on Top of Your ETL Pipelines With Table Freshness Checks
February 24, 2021Freshness checks are an easy way to see if your data is out of date. A freshness check queries a table for the MAX created time, and…
Golang - Detecting Resource Leaks With Baseline Tests
January 23, 2021Cleaning up resources is one fo the most difficult things in Go. If careful attention isn't paid to files, network connections, or…
Snowflake CSV COPY Into Examples
January 22, 2021This article lists a number snowflake configuration csv copy into parameters and the outputs they produce. I'm currently working on a…
Its Never Been Easier To Get Started With Data
November 22, 2020The low cost of storage combined with systems like Presto, which decouple computer from storage, makes it easier than ever to get started…
Introduction to How External Partitions Work in Systems Like Redshift Spectrum
November 07, 2020When I started using Spectrum and Athena I was confused by partitions. I found few resources and examples on how partitions actually worked…
2020 Summer of Sprouts
October 24, 2020Last summer I grew sprouts. I've always enjoyed gardening but am currently living in an apartment with very little light exposure. Due to…
How Probes Partition the Debug Space
August 30, 2020Probing is a technique to perform regular checks on a service using a short interval. Probes provide signals that can significantly cut down…
Understanding & Modifying Systems - Goals
August 02, 2020Identifying system boundaries is one of the hardest parts of understanding and modifying systems. A focus on goals acts as a compass by…
Understanding & Modifying Systems
July 31, 2020Systems exist all around us. Every day we operate in an incalculable number of systems. Our biological systems dictate our physical needs…
Do You Like Your Internal Customers? Give Them Self-Service!
July 15, 2020Self-service tools enable predictable delivery while still allowing for organizational oversight. Self-service tools allow customers to…