A DAG is just a Python file that defines tasks and their order. Write one, set dependencies, and watch the scheduler run it in the right sequence.
Why: you define a workflow as code — a DAG object with a schedule, containing tasks — which makes it reviewable, version-controlled, and testable like any other code. When: create one .py file per workflow in the dags/ folder. Where: the DAG’s parameters (schedule, start_date, retries) govern how and when it runs.
from airflow import DAG
from airflow.operators.bash import BashOperator
import pendulum
with DAG(
dag_id="hello_pipeline",
start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
schedule="@daily", # run once a day
catchup=False, # don't backfill history on first deploy
) as dag:
step = BashOperator(task_id="say_hello", bash_command="echo 'hello airflow'")Why: the value of a DAG is ordering — you declare which tasks depend on which, and Airflow runs them in that order, parallelizing where it can. When: use the >> operator to wire dependencies; a task runs only after all its upstreams succeed. Where: this dependency graph is what a plain script cannot express.
from airflow.operators.bash import BashOperator
ingest = BashOperator(task_id="ingest", bash_command="echo ingest")
validate = BashOperator(task_id="validate", bash_command="echo validate")
train = BashOperator(task_id="train", bash_command="echo train")
# Wire the order — train waits for validate, which waits for ingest:
ingest >> validate >> train
# Fan-out / fan-in also works:
# ingest >> [validate_a, validate_b] >> trainWhy: you should run a single task and a full DAG from the CLI before trusting the scheduler — it catches import errors and logic bugs fast, without waiting for a schedule. When: use airflow tasks test to run one task now, and airflow dags test to run the whole DAG for a date. Where: these run in-process and do not affect the scheduler’s state.
# List DAGs Airflow has discovered (catches import errors):
airflow dags list
# Run a single task for a logical date, printing logs to your terminal:
airflow tasks test hello_pipeline say_hello 2024-01-01
# Run the entire DAG once for a date:
airflow dags test hello_pipeline 2024-01-01