Most pipeline breaks come from upstream schema changes nobody warned you about. Data contracts make the interface explicit, and schema evolution rules let data change without breaking consumers.
A huge share of data incidents trace to a producer changing something — renaming a column, changing a type, dropping a field — with no warning to the pipelines downstream. Because the producer often does not even know who consumes their data, these breaks are silent until a dashboard goes blank. Data contracts exist to turn this implicit, fragile coupling into an explicit agreement.
The most common cause of a broken pipeline:
an app team renames user_id -> uid in their database
-> your ingestion still selects user_id -> nulls / job fails
-> a dashboard breaks -> you get paged -> you discover the change
The producer didn't know you existed. The interface between them and you
was IMPLICIT. A data contract makes it EXPLICIT and enforceable.A data contract is a versioned, machine-readable definition of a dataset’s interface: its fields and types, which are required, semantic meaning, quality guarantees (e.g. non-null keys), and an owner. It is the API contract idea applied to data. With it, the producer knows what they must not break, and the consumer knows what they can rely on.
# contracts/orders.yaml — the agreed interface for the orders dataset
dataset: orders
owner: checkout-team
version: 2
fields:
- name: order_id type: string required: true unique: true
- name: customer_id type: string required: true
- name: amount type: decimal required: true constraint: ">= 0"
- name: status type: string allowed: [placed, shipped, completed]
- name: ordered_at type: timestamp required: true
guarantees:
freshness: "< 1 hour"
completeness: "order_id, customer_id never null"
# the producer must not break this without a version bump + notice.A contract is only real if something checks it. Validate incoming data against the contract at ingestion — reject or quarantine data that violates it, and fail loudly on breaking changes. Ideally the check runs in the producer’s CI too, so a breaking change is caught before it ships rather than after it corrupts your warehouse.
# validate an incoming batch against the contract before it enters the lake
import yaml, json
contract = yaml.safe_load(open("contracts/orders.yaml"))
required = [f["name"] for f in contract["fields"] if f.get("required")]
def check(batch_columns: set):
missing = set(required) - batch_columns
if missing:
raise ValueError(f"Contract violation: missing required fields {missing}")
check(set(df.columns)) # fail ingestion on a breaking change
# best: run this in the PRODUCER's CI too, so breaks are caught pre-release.Data must still be allowed to change — the goal is safe change, not frozen schemas. The rule is backward compatibility: additive changes (new optional columns) are safe; breaking changes (rename, drop, retype a required field) require a version bump and a migration window. Formats like Avro/Protobuf and warehouse schema-evolution features enforce these rules so consumers tolerating unknown fields keep working.
Schema changes, classified:
SAFE (backward compatible) — consumers keep working:
+ add a new OPTIONAL column
+ add a new enum value (if consumers ignore unknowns)
widen a type (int -> long)
BREAKING — needs a version bump + notice + migration window:
- rename or drop a column
- make an optional field required
- change/narrow a type (string -> int)
Rule: producers may ADD freely; anything that could break a consumer is a
new contract version. Avro/Protobuf + a schema registry enforce this
automatically. Consumers should tolerate unknown fields.