A cron job and a script fall apart the moment a step fails or depends on another. An orchestrator runs multi-step workflows with dependencies, retries, and visibility.
Why: real ML workflows are multi-step (ingest → validate → train → evaluate → register) with dependencies and failure points that a bare script or cron cannot handle — an orchestrator runs the steps in order, retries failures, and shows you what happened. When: reach for one the moment a workflow has more than one step or must run on a schedule reliably. Where: Airflow is the most widely used; the concepts transfer to any orchestrator.
CRON + SCRIPT ORCHESTRATOR (Airflow)
one blob, no dependencies steps with explicit dependencies (a DAG)
a failure kills everything per-step retries + alerting
"did it run?" -> check logs UI shows every run and step status
no backfill schedule, catch up, and backfill historyWhy: Airflow models a workflow as a DAG (Directed Acyclic Graph) of tasks, where each task is an instance of an operator (a unit of work) and edges define order — grasp these three words and Airflow makes sense. When: think "DAG = the workflow, task = a step, operator = what the step does." Where: the scheduler walks the DAG and runs tasks when their upstream dependencies succeed.
DAG the whole workflow, a graph with no cycles
|
TASK one node = one step (an instance of an operator)
|
OPERATOR what a task actually does (run Python, run bash, call an API)
ingest >> validate >> train >> evaluate >> register
(each ">>" is a dependency edge; a step runs only after its parents succeed)Why: the fastest way to learn is a local standalone instance, which starts the scheduler and web UI together and auto-creates a login. When: use standalone to develop DAGs; a real deployment separates the scheduler, webserver, and workers. Where: DAG files go in the dags/ folder, and Airflow picks them up automatically.
pip install apache-airflow
export AIRFLOW_HOME=~/airflow
airflow standalone # starts scheduler + webserver, prints a login
# Open http://localhost:8080. Put your DAG .py files in $AIRFLOW_HOME/dags/
# and they appear in the UI automatically.