Spark processes data across a cluster of machines with one API. Understand the driver/executor model, why Spark is lazy, and the DataFrame abstraction that makes distributed computing feel like pandas.
Tools like pandas load all data into one machine’s memory — perfect until the data is larger than that machine. Spark solves this by splitting data into partitions spread across a cluster and running the same operation on each partition in parallel. You write code as if against one dataset; Spark distributes the execution. That is the entire value proposition: scale from a laptop to thousands of cores with the same API.
pandas: whole dataset in ONE machine's RAM
df = pd.read_csv("10TB.csv") # dies — doesn't fit
Spark: dataset split into PARTITIONS across many machines
10TB -> [2GB][2GB][2GB]... each partition processed in parallel
You write one transformation; Spark runs it on every partition at once,
across the cluster. Same code scales from 1 core to 10,000.A Spark application has one driver that holds your program and plans the work, and many executors across the cluster that do the actual computation on partitions. The driver breaks a job into tasks (one per partition), ships them to executors, and collects results. Understanding this split explains most Spark behavior — including why calling collect() can crash the driver by pulling all data back to it.
DRIVER (your program + the plan)
| splits the job into tasks (1 per partition)
v
+-------+--------+--------+
| EXEC | EXEC | EXEC | executors run tasks on partitions,
| p0,p1 | p2,p3 | p4,p5 | in parallel, holding data in memory
+-------+--------+--------+
The driver PLANS + coordinates; executors DO the work.
Danger: df.collect() pulls ALL partitions back to the driver's memory ->
crashes on big data. Keep data distributed; only collect small results.Spark operations come in two kinds. Transformations (select, filter, join, groupBy) are lazy — they build up a plan but do not run. Only an action (count, show, write, collect) triggers execution of the whole plan. This laziness lets Spark see the full pipeline and optimize it (via the Catalyst optimizer) before doing any work, e.g. pushing filters down to read less data.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("intro").getOrCreate()
df = spark.read.parquet("sales.parquet")
# TRANSFORMATIONS — lazy, nothing runs yet, just builds a plan:
big = (df.filter(df.amount > 100)
.select("customer_id", "amount")
.groupBy("customer_id").sum("amount"))
# ACTION — NOW Spark optimizes the whole plan and executes it:
big.show() # or .count(), .write..., .collect()
# Because it's lazy, Spark can push the filter into the read + prune
# columns BEFORE scanning — you get optimization for free.Spark’s original API was the RDD (a low-level distributed collection). Modern Spark centers on the DataFrame — a distributed table with named, typed columns — because it lets the Catalyst optimizer rewrite your query and run it far faster than hand-written RDD code. Unless you have a rare low-level need, you use DataFrames (and Spark SQL), which also read almost exactly like pandas.
RDD low-level distributed collection of objects.
you control everything; the optimizer can't help -> often slower.
rarely needed today.
DataFrame distributed TABLE: named, typed columns (like a pandas df,
but partitioned across the cluster).
-> Catalyst optimizer rewrites + speeds up your query
-> reads like pandas; interops with Spark SQL
Default to DataFrames. Reach for RDDs only for unusual low-level control.
The rest of this course is the DataFrame API.