A Spark job is only as good as its I/O. Read and write CSV, JSON, and Parquet; choose the right format; and write partitioned output so downstream queries stay fast.
Spark reads and writes many formats through a uniform API: spark.read.<format> and df.write.<format>. The same job can ingest CSV, emit Parquet, and load a database via JDBC. Write modes (overwrite, append, error, ignore) control what happens if the target exists — a detail that matters a great deal in pipelines that re-run.
# read various sources
csv = spark.read.option("header", True).csv("input/orders.csv")
js = spark.read.json("input/events.json")
pq = spark.read.parquet("input/sales.parquet")
jdbc = (spark.read.format("jdbc")
.option("url", "jdbc:postgresql://db/app")
.option("dbtable", "public.orders")
.option("user", "reader").load())
# write — mind the mode on re-runnable jobs:
csv.write.mode("overwrite").parquet("output/orders/")
# overwrite | append | error (default) | ignoreFormat choice dominates I/O performance. CSV and JSON are row-oriented, untyped, and uncompressed — fine as raw inputs, poor for repeated analytics. Parquet is columnar, typed, compressed, and carries statistics Spark uses to skip data (predicate pushdown). Converting raw data to Parquet once early in a pipeline speeds up everything downstream — the same lesson as the Data Modeling course’s lakehouse formats.
Format for analytical Spark jobs:
CSV / JSON row-oriented, untyped, uncompressed
-> Spark reads EVERY column + re-parses text each time. Slow.
Fine as a raw landing format, not for repeated reads.
PARQUET columnar, typed, compressed, min/max stats per block
-> reads only needed columns, skips blocks (pushdown),
~5-10x smaller. The default for lake/warehouse data.
Convert raw CSV -> Parquet ONCE at ingestion; every later job benefits.Writing output partitioned by a column (usually a date) lays the data out in folders so downstream readers with a matching filter skip everything else — partition pruning at the file-system level. This is the write-side complement to warehouse partitioning and one of the highest-impact choices for a pipeline that feeds other jobs.
# write partitioned by date -> folders like ordered_date=2024-01-01/
(clean.write
.mode("overwrite")
.partitionBy("ordered_date")
.parquet("lake/sales/"))
# a downstream reader filtering the partition column reads only those folders:
recent = spark.read.parquet("lake/sales/").filter("ordered_date >= '2024-06-01'")
# Spark prunes to matching partitions -> reads a fraction of the data.
# Watch partition cardinality: partitioning by a high-uniqueness column
# (e.g. user_id) creates millions of tiny files — a performance disaster.A common pipeline pathology is producing thousands of tiny output files (from too many partitions or highly granular partitionBy), which cripples downstream reads and overloads the file system’s metadata. The fix is to control the number of output files with coalesce or repartition before writing, targeting reasonably sized files (hundreds of MB). Managing file size is a routine data-engineering concern.
# too many output files kill downstream performance. control them:
(df.repartition(200) # ~200 output files (shuffles to rebalance)
.write.parquet("out/"))
(df.coalesce(10) # reduce to 10 files WITHOUT a full shuffle
.write.parquet("out/"))
# repartition(n): full shuffle, even-sized partitions (use to increase/rebalance)
# coalesce(n): merges partitions, no shuffle (use to REDUCE file count cheaply)
# Aim for files in the hundreds-of-MB range, not thousands of tiny ones.