Process data too big for one machine. The distributed execution model and lazy evaluation, the DataFrame API for transformations, Spark SQL and joins, reading and writing real file formats, the partitioning and shuffle mechanics that decide performance, and structured streaming for continuous pipelines — all in runnable PySpark.
Before you start
PySpark is Python plus SQL thinking. The Python and Data Analysis with Pandas courses are ideal preparation — the DataFrame API will feel familiar.
Why Spark & How It Runs
Spark processes data across a cluster of machines with one API. Understand the driver/executor model, why Spark is lazy, and the DataFrame abstraction that makes distributed computing feel like pandas.
DataFrames & Transformations
The DataFrame API is how you actually shape data in Spark — select, filter, add columns, and aggregate. It reads like pandas but runs distributed and lazy across the cluster.
Spark SQL & Joins
Spark runs SQL over DataFrames, and the two APIs are interchangeable. Learn to query with SQL, join datasets, and use broadcast joins to make small-to-large joins fast.
Reading & Writing Data
A Spark job is only as good as its I/O. Read and write CSV, JSON, and Parquet; choose the right format; and write partitioned output so downstream queries stay fast.
Partitioning, Shuffles & Performance
Spark performance is mostly about one thing: minimizing and balancing the shuffle. Understand partitions, what triggers a shuffle, data skew, and when to cache.
Structured Streaming
The same DataFrame API that processes a file can process an unbounded stream. Learn Spark Structured Streaming — reading a source continuously, windowed aggregations, watermarks, and exactly-once output.