When data outgrows one machine, Spark spreads the work across a cluster. Learn the DataFrame API, lazy evaluation, and the transformations you’ll use daily — in PySpark.
Why: pandas is fast and simple but lives in one machine’s memory — when data is tens of gigabytes to petabytes, Spark distributes both the data and the computation across a cluster, using the same DataFrame concepts you already know. When: reach for Spark when data no longer fits in memory or a single machine is too slow. Where: PySpark is the Python API; you write DataFrame code and Spark parallelizes it.
pip install pyspark # needs a Java runtime (JDK 11+) installedWhy: Spark DataFrames look like pandas but are lazy — transformations (select, filter, withColumn) build a plan and nothing runs until an action (show, count, write) triggers it — which lets Spark optimize the whole plan before executing. When: chain transformations freely; expect work only at the action. Where: this laziness is why Spark scales — it plans across the cluster before moving any data.
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("etl").getOrCreate()
df = spark.read.parquet("s3://lake/events/") # lazy — nothing read yet
# Transformations build a plan (still nothing runs):
clean = (df
.filter(F.col("event") == "purchase")
.withColumn("amount_usd", F.col("amount") * F.col("fx_rate")))
clean.show(5) # ACTION — now Spark reads, filters, and computesWhy: most Spark work is a handful of operations — select and filter rows, add columns, group and aggregate — expressed almost identically to SQL and pandas. When: use these to clean and reshape raw data into analytics-ready or ML-ready form. Where: writing the result back to partitioned Parquet completes the batch stage.
from pyspark.sql import functions as F
daily = (df
.filter(F.col("event") == "purchase")
.groupBy("user_id", F.to_date("ts").alias("dt"))
.agg(
F.count("*").alias("purchases"),
F.sum("amount").alias("total_spent"),
))
# Write partitioned Parquet back to the lake:
daily.write.partitionBy("dt").mode("overwrite").parquet("s3://lake/daily/")