Workflow Orchestration

After this skill you can…

• Build a DAG in Airflow with sensors and operators
• Implement backfilling and retries
• Set up monitoring and SLA alerts

Prerequisites

Watch (10 videos)

Introducing Lakehouse//RT and Reyden — Reynold Xin, Co–founder and Chief Architect

Databricks · beginner

→ Orchestrate workflows for real-time data processing→ Integrate Reyden with existing data pipelines

Learn ETL Pipelines in Databricks in Under 1 Hour | Data Engineering in Databricks

Alex The Analyst · beginner

→ Design Workflows→ Orchestrate Data Pipelines→ Automate Tasks

Lakeflow In Action (Gourmet Pipeline Demo)

Databricks · intermediate hands-on

→ Automate workflow orchestration→ Manage data pipelines

Iceberg is harder than you think

MLOps.community · intermediate hands-on

→ Orchestrate data workflows→ Optimize data processing

Data Engineering Roadmap (2026)

Coursera · intermediate

→ Orchestrate data workflows→ Manage data pipelines

Data Orchestration Explained #datascience

Decodo (formerly Smartproxy) · beginner hands-on

→ Automate data workflows→ Manage data pipelines→ Ensure correct workflow order

How to Become a Data Engineer? | Data Engineering Training | Edureka

edureka! · beginner

→ Orchestrate data workflows→ Manage data pipelines→ Integrate data tools

Data Triggers and Advanced Control Flow With Lakeflow Jobs

Databricks · intermediate hands-on

→ Design workflows→ Implement conditional logic→ Use looping and job modularity

Unified orchestration for any workload with Lakeflow Jobs - Data Engineering with Databricks

Databricks · beginner hands-on

→ Streamline data workflows→ Implement workflow automation

The Evolution of Lyft's Feature Store // Devon Mittow // DE4AI

MLOps.community · beginner

→ Orchestrate a data workflow→ Manage a feature store

Read (10 articles)