When seconds of freshness matter, data streams instead of batching. Kafka is the backbone — a durable log of events that producers write and consumers read independently.
Why: Kafka decouples systems that produce data from those that consume it — producers append events to a "topic" (a durable, ordered log), and any number of consumers read at their own pace — so one event stream can feed many pipelines. When: use it for real-time ingestion, event-driven systems, and CDC. Where: topics are split into partitions for parallelism, and consumers track their position (offset) so they can resume.
PRODUCERS ──► [ TOPIC: partitioned, durable, ordered log ] ──► CONSUMERS
web app part 0: e1 e2 e3 e4 ... fraud model
mobile part 1: e5 e6 e7 ... warehouse ETL
services feature store
Producers append; consumers read at their own offset. One stream,
many independent readers. Events are retained, not deleted on read.Why: the two halves of Kafka are dead simple — a producer sends messages to a topic, a consumer subscribes and receives them — and everything else (durability, ordering, replay) is Kafka doing its job. When: produce from the systems generating events; consume from the pipelines that need them. Where: keying messages sends related events to the same partition, preserving their order.
# pip install kafka-python
from kafka import KafkaProducer, KafkaConsumer
import json
# Producer — append events to a topic:
producer = KafkaProducer(
bootstrap_servers="localhost:9092",
value_serializer=lambda v: json.dumps(v).encode(),
)
producer.send("purchases", key=b"user-42",
value={"user_id": 42, "amount": 19.99})
producer.flush()
# Consumer — read them (from another process/service):
consumer = KafkaConsumer("purchases", bootstrap_servers="localhost:9092",
value_deserializer=lambda b: json.loads(b))
for msg in consumer:
print(msg.value) # {"user_id": 42, "amount": 19.99}Why: streaming data usually still needs to land in the lake/warehouse for training and analytics, so a consumer (or a stream processor like Spark Structured Streaming) reads the topic and writes batches to storage — bridging real-time and batch worlds. When: use streaming ingestion for freshness, then persist to Parquet for ML training. Where: this "stream in, batch at rest" pattern feeds both real-time features and offline training from one source.
# Spark Structured Streaming: read a Kafka topic, write to the lake.
from pyspark.sql import functions as F
stream = (spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "purchases")
.load())
parsed = stream.select(F.col("value").cast("string"))
(parsed.writeStream
.format("parquet")
.option("path", "s3://lake/purchases/")
.option("checkpointLocation", "s3://lake/_checkpoints/purchases/")
.start()) # continuously lands streamed events as Parquet