Operators are the building blocks — run bash, Python, or call a service. The modern TaskFlow API turns plain Python functions into tasks with almost no boilerplate.
Why: an operator is a reusable template for a kind of work — BashOperator runs a command, PythonOperator runs a function, and hundreds of provider operators talk to S3, databases, Spark, and cloud services — so you rarely write integration code from scratch. When: pick the operator that matches the task; install provider packages for external systems. Where: each operator instance becomes one task in the DAG.
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
def preprocess():
print("cleaning data...")
t1 = BashOperator(task_id="download", bash_command="curl -o data.csv https://...")
t2 = PythonOperator(task_id="preprocess", python_callable=preprocess)
t1 >> t2
# Provider operators exist for S3, Postgres, Spark, Kubernetes, GCP, AWS, ...Why: the modern TaskFlow API lets you decorate plain Python functions with @task and call them like functions — Airflow turns them into tasks and infers dependencies from how you pass their return values, eliminating most boilerplate. When: prefer TaskFlow for Python-centric pipelines (most ML work). Where: returning a value from one task and passing it to another creates the dependency automatically.
from airflow.decorators import dag, task
import pendulum
@dag(start_date=pendulum.datetime(2024, 1, 1, tz="UTC"),
schedule="@daily", catchup=False)
def etl():
@task
def extract():
return [1, 2, 3]
@task
def transform(nums):
return [n * 10 for n in nums]
@task
def load(nums):
print("loading", nums)
load(transform(extract())) # data flow = dependency graph
etl()Why: Airflow retries failed tasks and can re-run past dates, so a task MUST be idempotent — running it twice for the same date produces the same result, not duplicates. When: design every task to overwrite or upsert by the run’s logical date, never blindly append. Where: this is the single most important habit for reliable pipelines; a non-idempotent task corrupts data on every retry.
NON-IDEMPOTENT (bad) IDEMPOTENT (good)
append rows every run overwrite/upsert partition for {{ ds }}
retry -> duplicated data retry -> same result
backfill -> chaos backfill -> correct history
Key each task's output to the run's logical date (ds). Re-running is safe.