Model ML workflows as Airflow DAGs — write tasks and dependencies, schedule and backfill runs, pass data between tasks, and build a real training pipeline (ingest → train → evaluate → register). Ends with where Kubeflow and other orchestrators fit. Every lesson is runnable Python.
Before you start
You will run real DAGs with apache-airflow — airflow standalone starts everything locally. Comfort with Python helps, and the MLflow & DVC course pairs naturally — the pipeline here tracks and registers with them.
Why Orchestration & Airflow Concepts
A cron job and a script fall apart the moment a step fails or depends on another. An orchestrator runs multi-step workflows with dependencies, retries, and visibility.
Your First DAG
A DAG is just a Python file that defines tasks and their order. Write one, set dependencies, and watch the scheduler run it in the right sequence.
Operators & the TaskFlow API
Operators are the building blocks — run bash, Python, or call a service. The modern TaskFlow API turns plain Python functions into tasks with almost no boilerplate.
Scheduling & Backfill
Airflow’s superpower is time. Schedule DAGs, understand the logical date, and backfill historical runs — the features a cron job can never give you.
Passing Data Between Tasks
Tasks run in separate processes, so they can’t share variables. XComs pass small values between tasks; for real data, pass references to shared storage.
An ML Training Pipeline
Put it together: a DAG that ingests data, validates it, trains a model, evaluates against a baseline, and registers the winner — the pipeline the MLOps checklist describes.
Kubeflow & Choosing an Orchestrator
Airflow is general-purpose; Kubeflow Pipelines is Kubernetes-native and ML-specific. Understand the trade-offs and pick the right orchestrator for your stack.