📰 Medium · Python

Singer taps are the extract half of an open-source contract that decides whether moving a new SaaS API or a Postgres table into your… Continue reading on Medium

Databricks has been moving data engineering toward a declarative model. Lakeflow Declarative Pipelines is the current framework for… Continue reading on Medium

Databricks was founded in 2013 by the original creators of Apache Spark at UC Berkeley’s AMPLab. The founders open-sourced Spark and then… Continue reading on M

llm evaluation for data pipelines is the load-bearing correctness discipline of the 2026 data stack — the difference between a RAG chatbot… Continue reading on

Subtitle: From “hope and pray” to “validate early, validate often” — how to catch bad data before it reaches your CEO’s dashboard. Continue reading on Medium »

Most production data pipelines do not fail because their underlying mathematical algorithms are flawed they implode because human domain… Continue reading on IL

A practical, question-driven guide with concise answers, exam traps, and a last-minute cheat sheet Continue reading on Medium »

I have been building a project to expand my understanding of dimensional modelling, DBT, Airflow, data engineering concepts and tools at… Continue reading on Me

If you’ve ever worked with a data lake, you probably know the pain. Files everywhere, half-finished writes, someone asks “can we just… Continue reading on Mediu

In this project, I created an updated version of my previous project. Although I didn’t use them in the previous one, this time I used… Continue reading on Medi

Data has become the driving force behind almost every modern business. Whether you’re shopping online, scrolling through social media… Continue reading on Mediu

windmill is the pick-one architectural decision that finally collapses "we need an orchestrator and an internal-tool builder and a secret… Continue reading on M

Building a Production-Oriented Python Library for Data Engineering Continue reading on Medium »

How I designed and built an end-to-end data engineering system that any team could inherit, run, and extend — using Python, DuckDB, Apache… Continue reading on

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

Designing a production-ready data ingestion pipeline with validation, analytics, testing, and cloud deployment. Continue reading on Medium »

If you’re getting into data engineering, you’ve probably heard the term “Airflow” thrown around a lot. Continue reading on Medium »

Reduce cloud data costs by 30–60% — optimize S3 storage tiers, Snowflake auto-suspend, spot instances, reserved capacity, partition… Continue reading on Medium

Introduction Continue reading on Medium »

Your scheduled scraper re-downloaded the whole corpus last night. A few thousand records. About forty of them actually changed since the… Continue reading on Me

Five Snowflake patterns that quietly kill your external ETL — and the gotchas nobody warns you about. Continue reading on Medium »

Workflow orchestration is one of the core requirements in almost every data engineering role. Continue reading on Medium »

Silent data failures are the worst kind. Here’s a system that makes them impossible Continue reading on Medium »

A field guide for data engineers who've hit the 404/EntityNotFound wall Continue reading on Medium »