The data plumbing every ML system rests on — pipelines and batch vs. streaming, data lakes and warehouses, ingestion patterns, batch processing with Spark, streaming with Kafka, and assembling it all into a pipeline that feeds a model. Scoped to what an MLOps engineer needs. Every lesson is runnable Python.
Before you start
You will run real Python. The Spark lessons use pyspark (which needs a Java 11+ runtime) and the streaming lesson uses kafka-python. Comfort with pandas from the Data Analysis course helps.
What Is Data Engineering?
Models are only as good as the data feeding them. Data engineering is the plumbing — moving and shaping data reliably. Meet pipelines, batch vs. streaming, and ETL vs. ELT.
Data Lakes & Warehouses
Where does all the data live? A lake stores raw files cheaply; a warehouse stores structured tables for fast queries; a lakehouse blends both. Plus the file format that matters.
Data Ingestion
Ingestion is how data enters your platform — pulled in batches, streamed as events, or captured from a database’s change log. Get it reliable and idempotent.
Batch Processing with Spark
When data outgrows one machine, Spark spreads the work across a cluster. Learn the DataFrame API, lazy evaluation, and the transformations you’ll use daily — in PySpark.
Spark at Scale
Joins and shuffles are where big-data jobs get slow or fall over. Understand what a shuffle costs, join efficiently, and write output that stays fast to read.
Streaming with Kafka
When seconds of freshness matter, data streams instead of batching. Kafka is the backbone — a durable log of events that producers write and consumers read independently.
A Data Pipeline for ML
Assemble the pieces into the pipeline that actually feeds a model: ingest to the lake, transform with Spark, build a feature table, and schedule it — the MLOps data backbone.