Spark performance is mostly about one thing: minimizing and balancing the shuffle. Understand partitions, what triggers a shuffle, data skew, and when to cache.
A DataFrame is divided into partitions, and each partition is processed by one task on one core. Too few partitions underuses the cluster; too many creates scheduling overhead. The number of partitions after a shuffle is controlled by a config, and matching it to your cluster size is basic tuning. Everything about Spark performance comes back to how data is partitioned.
DataFrame = N partitions. Each partition -> one task -> one core.
too FEW partitions -> idle cores, giant tasks, out-of-memory
too MANY partitions -> scheduling overhead, tiny wasteful tasks
goal: partitions ~= a small multiple of total cluster cores
# post-shuffle partition count (default 200 — often wrong for your cluster):
spark.conf.set("spark.sql.shuffle.partitions", 400)
df.rdd.getNumPartitions() # inspect current partitioningA shuffle moves data across the network so related rows (same join/group key) end up on the same executor. It is triggered by groupBy, join, distinct, and repartition, and it is by far the costliest operation because it writes to disk and crosses the network. Most Spark optimization is really “do fewer and smaller shuffles”: filter early, broadcast small joins, avoid needless repartitions.
Wide transformations = SHUFFLE (network + disk, expensive):
groupBy, join, distinct, repartition, orderBy
Narrow transformations = NO shuffle (cheap, stay in place):
select, filter, withColumn, map
Performance playbook = minimize shuffle:
- filter/select EARLY so less data moves
- broadcast() small-side joins (previous lesson)
- set spark.sql.shuffle.partitions to fit your cluster
- don't repartition without a reasonEven with the right partition count, performance dies if one key holds most of the rows — a single executor gets a giant partition while others idle. This skew is a top real-world Spark problem (e.g. a null key, or one huge customer). You diagnose it from the Spark UI (one task runs far longer) and fix it by salting the key or enabling adaptive skew handling.
# symptom: 199 tasks finish fast, 1 runs for an hour (Spark UI shows it).
# cause: one key (e.g. customer_id = NULL or a mega-customer) has most rows.
# fix 1: let Spark handle it (Spark 3+ adaptive execution)
spark.conf.set("spark.sql.adaptive.enabled", True)
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True)
# fix 2: "salt" the hot key so it spreads across partitions
from pyspark.sql import functions as F
salted = df.withColumn("salt", (F.rand() * 10).cast("int"))
# join on (key, salt) to split the skewed key into 10 sub-partitions.By default Spark recomputes a DataFrame every time an action touches it, because it holds a plan, not results. If you use the same intermediate DataFrame several times (loops, multiple outputs), caching it in memory avoids recomputation. But caching costs memory and helps nothing if the data is used once — so cache deliberately, and unpersist when done.
# expensive DataFrame used by several downstream actions -> cache once:
active = df.filter("status = 'active'").join(broadcast(dim), "id")
active.cache() # or .persist(StorageLevel.MEMORY_AND_DISK)
active.count() # first action: computes AND caches
report_a = active.groupBy("region").count() # reuse cache — no recompute
report_b = active.groupBy("segment").sum("amount")
active.unpersist() # free the memory when done
# DON'T cache data used only once — it just wastes memory. Cache = reuse.