Workflow Orchestration
Schedule and monitor data pipelines with Airflow, Dagster, or Prefect.
0%
Confidence · no data yet
After this skill you can…
- Build a DAG in Airflow with sensors and operators
- Implement backfilling and retries
- Set up monitoring and SLA alerts
Prerequisites
Watch (10 videos)
Data Triggers and Advanced Control Flow With Lakeflow Jobs
→ Design workflows→ Implement conditional logic→ Use looping and job modularity
Designing ETL Pipelines with Medallion Architecture in Azure Synapse
→ Structure ETL pipelines across Bronze, Silver, and Gold layers
Dagster vs Airflow vs Prefect | Which Workflow Platform is BEST in 2025?
→ Orchestrate data workflows with Dagster→ Compare Airflow and Prefect for workflow management
AWS Dev Day - Data & Analytics | Amazon Managed Airflow or AWS Step functions? (Hebrew)
→ Orchestrate workflows with AWS Step Functions→ Use Amazon Managed Airflow for data pipelines
Orchestrating Spark Jobs with Kubeflow // Sadik Bakiu // MLOps Meetup #71
→ Orchestrate Spark jobs with Kubeflow→ Integrate Apache Spark with Kubernetes
Orchestrating Machine Learning Workflows with Prefect // Kevin Kho // MLOps Meetup #94
→ Build a Prefect workflow for ML pipelines→ Orchestrate a hyperparameter grid search with Dask
Unified orchestration for any workload with Lakeflow Jobs - Data Engineering with Databricks
→ Streamline data workflows→ Implement workflow automation
Orchestration With Lakeflow Jobs
→ Orchestrate data pipelines→ Manage workflows→ Automate tasks
Lakeflow In Action (Gourmet Pipeline Demo)
→ Automate workflow orchestration→ Manage data pipelines
Job Orchestration in Databricks | Data Engineering in Databricks
→ Orchestrate jobs in Databricks→ Automate data pipelines with Spark
DeepCamp AI