Airflow is general-purpose; Kubeflow Pipelines is Kubernetes-native and ML-specific. Understand the trade-offs and pick the right orchestrator for your stack.
Why: Kubeflow Pipelines is built for ML on Kubernetes — each step runs as a container in the cluster, with native experiment tracking and artifact lineage — so it suits teams already standardized on Kubernetes for ML. When: choose it when your training runs in K8s and you want per-step containers and GPU scheduling handled natively. Where: you define pipelines in Python too, but each component is a containerized step.
# pip install kfp — Kubeflow Pipelines SDK (each step is a container).
from kfp import dsl
@dsl.component
def train(data_path: str) -> float:
# runs as its own container in the cluster
return 0.91
@dsl.pipeline(name="training-pipeline")
def pipeline(data_path: str):
train(data_path=data_path)
# Compiled to a spec and run on a Kubeflow cluster, each step in its own pod.Why: there is no single best orchestrator — Airflow is general and mature, Kubeflow is K8s-native and ML-specific, and newer tools (Prefect, Dagster, Metaflow) trade differently on ergonomics and data-awareness — so the choice depends on your stack and team. When: match the tool to where your compute lives and what your team already runs. Where: the concepts you learned (DAGs, tasks, idempotency, scheduling) transfer to all of them.
Airflow general-purpose, huge ecosystem, mature; not ML-specific
Kubeflow Kubernetes-native, per-step containers, ML lineage built in
Prefect Pythonic, dynamic workflows, nice local dev
Dagster asset/data-aware, strong typing and testing
Metaflow data-science-first (Netflix), easy scaling
Pick by: where compute runs (K8s?), team familiarity, ML-native needs.
The DAG concepts carry over to all of them.Why: the orchestrator is the conductor that ties the rest of the MLOps stack together — it triggers training, calls MLflow to track and register, versions data with DVC, and kicks off deployment — so it sits at the center of the workflow. When: let the orchestrator own the end-to-end flow rather than gluing scripts by hand. Where: this closes the loop with the other courses — tracking, registry, and versioning become steps the DAG runs.
ORCHESTRATOR (Airflow/Kubeflow) drives the whole loop:
trigger ─► ingest+version data (DVC) ─► train + track (MLflow)
▲ │
│ ▼
monitor/drift ◄── serve ◄── register + promote (MLflow registry)
The orchestrator is the conductor; the other tools are the instruments.