Models are only as good as the data feeding them. Data engineering is the plumbing — moving and shaping data reliably. Meet pipelines, batch vs. streaming, and ETL vs. ELT.
Why: a model is a small part of a real ML system — most of the work is getting clean, timely, correctly-shaped data to it, and that plumbing is data engineering. When: it comes before modeling and never stops, since production models need a continuous supply of fresh data. Where: an MLOps engineer owns the reliability of this data flow, not just the model.
A production ML system, by effort:
data collection + engineering ████████████████ (most of it)
the model ██
serving + monitoring ████
"Garbage in, garbage out" is a data-engineering problem.
Reliable data > a fancier model.Why: data engineering is built from pipelines — a series of stages that move data from a source to a usable destination, transforming it along the way — the same shape whether it runs once or every minute. When: build a pipeline whenever data must flow from A to B repeatably. Where: each stage should be idempotent so a re-run is safe, exactly like the orchestration and preprocessing courses.
SOURCE ─► INGEST ─► STORE ─► TRANSFORM ─► SERVE
(DB, API, (pull/ (lake/ (clean, (feature table,
files, stream) warehouse) join, agg) model, dashboard)
events)
Each stage idempotent + observable. A pipeline is the unit of data work.Why: two fundamental choices shape every pipeline — batch (process data in chunks on a schedule) vs. streaming (process events as they arrive), and ETL (transform before loading) vs. ELT (load raw, transform in the warehouse) — and picking right depends on latency and cost. When: batch for most ML training data; streaming when freshness is critical (fraud, recommendations). Where: modern cloud favors ELT — land raw data cheaply, transform with warehouse compute.
BATCH process chunks on a schedule high throughput, minutes-hours
STREAMING process events as they arrive low latency, seconds
ETL transform -> load (classic; transform before the warehouse)
ELT load -> transform (modern; land raw, transform in-warehouse)
Most ML training data is BATCH + ELT. Reach for streaming when
seconds of freshness actually change the outcome.