When a number looks wrong, the first question is "where did it come from?" Data lineage traces that path, and a data catalog makes datasets discoverable and understandable.
Data lineage is the record of how data flows and transforms from source to dashboard — which tables feed which, through which jobs. It answers the two questions you ask under pressure: for an incident, "what downstream is affected by this bad source?" (impact analysis), and for a suspicious number, "what upstream produced it?" (root-cause). Without lineage, every investigation is archaeology.
Lineage = the dependency graph of your data:
shop_db.orders --> stg_orders --> fct_orders --> revenue_dashboard
shop_db.users --> stg_users --/ \-> finance_export
Two questions it answers instantly:
IMPACT "shop_db.orders is corrupt — what's downstream?"
-> stg_orders, fct_orders, both consumers. Go warn them.
ROOT CAUSE "revenue looks wrong — what feeds it?"
-> trace UP to the source and find where it broke.
Without lineage, both are hours of manual archaeology.You rarely draw lineage by hand — it is captured automatically. dbt derives it from ref()/source() (see the dbt course); orchestrators and query logs reveal table-to-table dependencies; open standards like OpenLineage let tools emit lineage events into a central catalog. The engineering goal is to make lineage a byproduct of running pipelines, not a document that rots.
Sources of automatic lineage:
dbt ref()/source() -> exact model-to-model DAG (free)
orchestrators Airflow/Dagster task deps + I/O
query logs parse warehouse SQL to infer table -> table edges
OpenLineage open standard: jobs emit lineage events to a catalog
(Marquez, DataHub, OpenMetadata, Unity Catalog)
Goal: lineage is a BYPRODUCT of running pipelines, captured as they run —
never a hand-drawn diagram that's out of date by Tuesday.A data catalog is the searchable inventory of an organization’s data: every dataset with its schema, owner, description, freshness, quality status, and lineage. It solves discovery ("does a table for this already exist?") and understanding ("what does this column mean, can I trust it?"). At scale, a catalog is what lets a company reuse data instead of rebuilding the same pipeline five times.
A data catalog entry (DataHub / OpenMetadata / Unity Catalog / Amundsen):
dataset: analytics.fct_orders
owner: data-platform-team
description: "One row per order. Grain: order."
schema: order_id, customer_id, amount, status, ordered_at
freshness: updated 12 min ago
quality: 4/4 tests passing
lineage: <- stg_orders <- shop_db.orders ; -> revenue_dashboard
popularity: queried by 34 people this week
Solves DISCOVERY ("does this exist?") + TRUST ("can I rely on it?").Metadata — data about the data — is what powers catalogs, lineage, and quality dashboards. Managing it means consistently capturing ownership, definitions, classifications (e.g. "contains PII"), and freshness across every dataset, ideally emitted automatically by pipelines. Good metadata turns a sprawling data platform from an unknowable mess into a navigable, governable system.
Types of metadata to capture (automatically, per dataset):
technical schema, types, partitions, size, row counts
operational freshness, last run, job that produced it, SLAs
business owner, description, definitions, business terms
governance classification (PII/PHI/public), retention, access policy
social who uses it, popularity, ratings
Emit it from pipelines into the catalog so it stays current. Rich, fresh
metadata is what makes a big data platform navigable AND governable
(next lesson).