Tasks run in separate processes, so they can’t share variables. XComs pass small values between tasks; for real data, pass references to shared storage.
Why: tasks are isolated, so to pass a value from one to another Airflow uses XComs (cross-communications) — small pieces of data stored in the metadata database. When: use XComs for lightweight results: a row count, a file path, a model’s metric. Where: with the TaskFlow API, returning a value and passing it to another task uses XComs automatically.
from airflow.decorators import dag, task
@task
def train():
accuracy = 0.91
return {"accuracy": accuracy, "model_path": "s3://models/run123/model.pkl"}
@task
def decide(result):
if result["accuracy"] > 0.85:
print("promote:", result["model_path"])
else:
print("reject — below threshold")
decide(train()) # the returned dict travels between tasks as an XComWhy: XComs live in the metadata database and are meant for small values — pushing a DataFrame or a model through them bloats the DB and slows Airflow — so pass a REFERENCE (an S3 path) and let tasks read the real data from shared storage. When: any time the payload is more than a few kilobytes. Where: the pattern is "task writes data to storage, returns the path; next task reads the path."
SMALL value (metric, path, count) -> XCom (fine)
BIG data (DataFrame, model, images) -> write to S3/GCS, pass the PATH
train(): save model to s3://.../model.pkl, RETURN the path
deploy(): RECEIVE the path, load from S3
Never push gigabytes through XComs — it hammers the metadata DB.Why: pipelines need configuration — Airflow Variables hold shared settings, and DAG params let you pass values when triggering a run manually — so you change behavior without editing code. When: use Variables for environment config (bucket names, thresholds), params for run-specific overrides. Where: keep secrets in a secrets backend, not in plain Variables.
from airflow.models import Variable
# A shared, UI/CLI-editable setting (not a secret):
bucket = Variable.get("model_bucket", default_var="s3://default-bucket")
# Trigger a run with params:
# airflow dags trigger ml_pipeline --conf '{"model_type": "xgboost"}'
# Inside a task, read it from the context: context["params"]["model_type"]
# Secrets belong in a secrets backend (Vault, AWS Secrets Manager), not Variables.