Data tests catch bad data before it reaches dashboards. Assert expectations in the pipeline, quarantine bad rows instead of dropping them, and place tests at the right points in the flow.
Just as unit tests assert on code, data tests assert on data: expected columns exist, values fall in range, keys are unique. Tools like Great Expectations (Python) and dbt tests (SQL) let you declare these expectations and run them in the pipeline. A failing expectation stops bad data from propagating — the data equivalent of a failing CI check.
# Great Expectations — declare what "good" looks like, then validate
import great_expectations as gx
batch = gx.read_parquet("orders.parquet")
batch.expect_column_values_to_not_be_null("order_id")
batch.expect_column_values_to_be_unique("order_id")
batch.expect_column_values_to_be_between("amount", min_value=0, max_value=100000)
batch.expect_column_values_to_be_in_set("status", ["placed", "shipped", "done"])
result = batch.validate()
if not result.success:
raise ValueError("Data quality checks failed — stopping the pipeline")
# (dbt tests express the same assertions in YAML/SQL — see the dbt course.)When rows fail validation, the worst response is to silently discard them — data vanishes and nobody knows why numbers are low. Instead, route failing rows to a quarantine table with the reason, let good rows proceed, and alert on the quarantine volume. This keeps the pipeline flowing while preserving evidence and making bad data visible and recoverable.
from pyspark.sql import functions as F
# split, don't drop: valid rows continue; invalid rows are kept with a reason
is_valid = (F.col("amount") > 0) & F.col("customer_id").isNotNull()
good = df.filter(is_valid)
bad = (df.filter(~is_valid)
.withColumn("quarantined_at", F.current_timestamp())
.withColumn("reason", F.lit("amount<=0 or null customer")))
good.write.mode("append").parquet("lake/silver/orders/")
bad.write.mode("append").parquet("lake/quarantine/orders/") # investigate + replay
# alert if quarantine volume spikes — silently dropping rows hides real bugs.Testing at the right points catches problems early and cheaply. Validate at ingestion (does the source match its contract?), after transformation (did the logic preserve correctness?), and before serving (is the mart safe to publish?). This mirrors the medallion layers: gate the bronze→silver and silver→gold transitions so bad data cannot climb toward consumers.
Place checks at each layer boundary (medallion architecture):
source --[check: matches contract?]--> BRONZE (raw)
--[check: types, nulls, ranges]--> SILVER (clean)
--[check: business rules, totals]--> GOLD (marts) --> BI
Fail EARLY: catching a schema break at ingestion is cheap; discovering it
in a CEO's dashboard is expensive. Gate each transition so bad data can't
climb toward consumers. Same shift-left idea as the DevSecOps pipeline.A quality check only helps if a human learns about a failure with enough context to act. Wire check results to alerts that name the dataset, the failed expectation, and the magnitude, and link a runbook. Track results over time so you can see a dimension slowly degrading before it becomes an incident. Silent checks are as useless as no checks.
A failed check must produce an ACTIONABLE alert:
❌ orders.silver — expect_amount_between failed
3.2% of rows (41,203) have amount <= 0 (threshold 0.5%)
first seen: 14:05 UTC upstream source: shop_db.orders
runbook: wiki/dq/orders-amount
Route to the owning team (not a firehose channel), include dataset +
expectation + magnitude + a link, and STORE every result so you can chart
a dimension degrading over weeks — not just today's pass/fail.