The same DataFrame API that processes a file can process an unbounded stream. Learn Spark Structured Streaming — reading a source continuously, windowed aggregations, watermarks, and exactly-once output.
Structured Streaming’s core idea is that a stream is just a table that grows forever, and you query it with the same DataFrame operations as a static table. Spark runs your query incrementally as new data arrives, maintaining results continuously. This unifies batch and streaming: the code is nearly identical, so you learn one API for both.
Batch: a fixed table -> run query once -> result.
Streaming: an UNBOUNDED table that keeps growing -> Spark re-runs the
query incrementally as new rows arrive -> continuously updated result.
Same DataFrame API for both. The only differences:
- read/write use readStream / writeStream
- you add time semantics (windows, watermarks) for event-time logic
Learn the batch API (previous lessons) and streaming is a small delta.You start a stream with readStream from a source (Kafka, a file directory, a socket), apply normal transformations, and end with writeStream to a sink, specifying an output mode and a checkpoint location. The checkpoint is what makes the stream fault-tolerant and exactly-once — it records progress so a restart resumes correctly. The Data Engineering course covers Kafka itself.
# read a Kafka topic as a streaming DataFrame
events = (spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "orders")
.load())
parsed = events.selectExpr("CAST(value AS STRING) as json") # then parse JSON
query = (parsed.writeStream
.format("parquet")
.option("path", "lake/orders/")
.option("checkpointLocation", "chk/orders/") # <- fault tolerance + exactly-once
.outputMode("append")
.trigger(processingTime="1 minute") # micro-batch every minute
.start())
query.awaitTermination()Streaming analytics usually aggregate over time windows — “orders per 5 minutes.” The catch is late data: events arrive out of order, so Spark must know how long to wait before finalizing a window. A watermark sets that bound (“accept data up to 10 minutes late”), letting Spark emit results and drop state for old windows so memory stays bounded. Windows plus watermarks are the heart of stream processing.
from pyspark.sql import functions as F
per_window = (parsed
.withWatermark("event_time", "10 minutes") # tolerate 10 min lateness
.groupBy(
F.window("event_time", "5 minutes"), # 5-min tumbling windows
"product_id")
.agg(F.count("*").alias("orders")))
# the watermark lets Spark:
# - still count events up to 10 min late into their window
# - then FINALIZE the window + drop its state (bounded memory)
# without a watermark, streaming aggregation state grows forever.Production streams must survive restarts without losing or duplicating data. Structured Streaming provides exactly-once semantics when the source is replayable (like Kafka) and the sink is idempotent, coordinated through the checkpoint that tracks exactly which data has been processed. Understanding this is what separates a demo stream from a pipeline you can trust with money.
Exactly-once in Structured Streaming needs three things:
1. REPLAYABLE source Kafka/file source can re-read after a crash
(tracks offsets)
2. CHECKPOINT records exactly what's been processed, so a
restart resumes at the right offset
3. IDEMPOTENT sink re-writing the same batch has no double effect
(file sink + checkpoint handles this)
Keep the checkpointLocation stable + durable (object storage). Delete it
and the stream forgets its progress -> reprocessing or gaps. It is the
stream's source of truth.