📰 Dev.to · Nithyalakshmi Kamalakkannan

Articles from Dev.to · Nithyalakshmi Kamalakkannan · 9 articles · Updated every 3 hours · View all reads

All ⚡ AI Lessons (11494) ArXiv cs.AI Dev.to · FORUM WEB Dev.to AI Forbes Innovation OpenAI News Hugging Face Blog

Part 8: Databricks Pipeline & Dashboard

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 8: Databricks Pipeline & Dashboard

Pipeline creation Databricks workflow is created with each task doing each part discussed...

Part 7: Gold Layer – Metrics, Watermarks, and Aggregations

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 7: Gold Layer – Metrics, Watermarks, and Aggregations

Gold tables answer business questions directly. Examples: Trips per hour by region Revenue per...

Part 6: Silver Layer – Cleansing, Enrichment, and Dimensions

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 6: Silver Layer – Cleansing, Enrichment, and Dimensions

The Silver layer converts raw events into analytics-ready records by: Cleaning bad data Enforcing...

Part 5: Building a ZIP Code Dimension Table

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 5: Building a ZIP Code Dimension Table

Why?, The Need for it! Fact tables (like taxi trips) are optimized for events: Pickup...

Part 4: Building the Bronze Layer with Auto Loader and Delta Lake

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 4: Building the Bronze Layer with Auto Loader and Delta Lake

The Bronze layer is the foundation of the entire streaming architecture. Its role is to ingest data...

Part 3: Simulating Real-Time Streaming Data Using Databricks Sample Datasets

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 3: Simulating Real-Time Streaming Data Using Databricks Sample Datasets

We use the Databricks NYC Taxi sample dataset, available by default in Databricks. This dataset is...

Part 2: Project Architecture

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 2: Project Architecture

The goal is not just to “make streaming work”, but to design a maintainable and observable streaming...

Part 1: Creating Databricks Workspace and Enabling Unity Catalog

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

Part 1: Creating Databricks Workspace and Enabling Unity Catalog

In Databricks, a secure, governed foundation for our data platform is provided by Unity Catalog,...

End-to-End Real-Time Data Engineering on Databricks Using Spark Structured Streaming and Delta Lake

Dev.to · Nithyalakshmi Kamalakkannan 3mo ago

End-to-End Real-Time Data Engineering on Databricks Using Spark Structured Streaming and Delta Lake

Simple batch processing and static dashboards have retired! Data platforms must ingest continuously...