Articles

Workflows just got even more powerful Continue reading on Medium »

If you’ve ever worked with a data lake, you probably know the pain. Files everywhere, half-finished writes, someone asks “can we just… Continue reading on Mediu

In this project, I created an updated version of my previous project. Although I didn’t use them in the previous one, this time I used… Continue reading on Medi

Data has become the driving force behind almost every modern business. Whether you’re shopping online, scrolling through social media… Continue reading on Mediu

windmill is the pick-one architectural decision that finally collapses "we need an orchestrator and an internal-tool builder and a secret… Continue reading on M

How I designed and built an end-to-end data engineering system that any team could inherit, run, and extend — using Python, DuckDB, Apache… Continue reading on

From Experience to Understanding: What Two Databricks Learning Tracks Taught Me About the Modern Data Lifecycle Continue reading on Medium »

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

How a simple command can save hours of debugging, reduce CI failures, and improve your team’s code quality. Continue reading on Medium »

Designing a production-ready data ingestion pipeline with validation, analytics, testing, and cloud deployment. Continue reading on Medium »

The moment I stopped worrying about the destination and started appreciating the journey Continue reading on Medium »

Three months into my first role, I found an uncomfortable reality: I didn’t really control the data I depended on daily. A data engineer… Continue reading on Me

Traditional batch-first data pipelines introduce artificial delays in data availability, forcing enterprise decisions to be made on stale information. This arti

If you’re getting into data engineering, you’ve probably heard the term “Airflow” thrown around a lot. Continue reading on Medium »

For years, data engineering and software engineering have lived in parallel worlds. Application developers shipped code through structured… Continue reading on

I offer 1:1 mentoring and interview prep for data engineers on Topmate. Now let’s get into it. Continue reading on Medium »

Vinay Chella and Akshat Goel discuss the challenges of running traditional CDC across heterogeneous databases during peak order traffic. They explain how Debezi

Distributed databases represent one of the most significant architectural paradigms in contemporary data engineering. After nine years… Continue reading on Medi

How Flink uses checkpoint barriers to take consistent snapshots of your running job so a crashed pod becomes a minor blip, not a… Continue reading on Towards Da

Why we’re open-sourcing State-Aware Orchestration for dbt in Orchestra and dbt core Continue reading on Medium »

The exact patterns that decide most data engineering SQL screens, with explanations that build real intuition. Continue reading on Medium »

Initially, organizations relied heavily on data warehouses, storing structured data in traditional databases like MySQL and building… Continue reading on Medium

Your scheduled scraper re-downloaded the whole corpus last night. A few thousand records. About forty of them actually changed since the… Continue reading on Me

Five Snowflake patterns that quietly kill your external ETL — and the gotchas nobody warns you about. Continue reading on Medium »