Airflow’s superpower is time. Schedule DAGs, understand the logical date, and backfill historical runs — the features a cron job can never give you.
Why: Airflow runs a DAG for a logical date (the period being processed), which is subtly different from wall-clock time — a daily DAG for 2024-01-01 typically runs just after that day ends, processing that day’s data. When: use the logical date (ds) to select the data window, not datetime.now(). Where: this is what makes runs deterministic and backfillable.
from airflow.decorators import dag, task
import pendulum
@dag(start_date=pendulum.datetime(2024, 1, 1, tz="UTC"), schedule="@daily")
def daily_report():
@task
def process(ds=None):
# ds is the logical date string, e.g. "2024-01-01" — use it to
# pick the data window, NOT datetime.now(). This makes reruns exact.
print(f"processing data for {ds}")
process()
daily_report()Why: when you deploy a DAG with a start_date in the past, catchup controls whether Airflow runs all the missed intervals — and backfill lets you deliberately re-run a historical range, which cron simply cannot do. When: set catchup=False to avoid a flood of runs on deploy; use backfill to (re)process history intentionally. Where: backfill only works correctly because your tasks are idempotent and keyed to the logical date.
# catchup=False in the DAG avoids auto-running every past interval on deploy.
# Deliberately (re)process a historical date range:
airflow dags backfill daily_report \
--start-date 2024-01-01 --end-date 2024-01-07
# Because tasks are idempotent + keyed to ds, backfilling reproduces
# correct history instead of duplicating or corrupting data.Why: production DAGs must recover from transient failures and tell you when something is wrong — per-task retries with backoff handle blips, and alerts/SLAs surface real problems. When: set retries and retry_delay on tasks; wire failure callbacks or email/Slack alerts. Where: an SLA miss (a task running longer than expected) is often the earliest signal of an upstream data problem.
from datetime import timedelta
from airflow.decorators import task
@task(retries=3, retry_delay=timedelta(minutes=5))
def flaky_api_call():
# Retried up to 3 times with a 5-minute delay on transient failures.
...
# DAG-level defaults + alerting:
# default_args={"retries": 2, "retry_delay": timedelta(minutes=5),
# "on_failure_callback": notify_slack}
# sla=timedelta(hours=1) -> alert if a task runs longer than expected