Not all data fits neatly in a warehouse. Learn the data lake, the file formats that make it queryable, the lakehouse table formats (Delta, Iceberg) that add reliability, and how the medallion architecture organizes raw to refined.
A warehouse stores structured, modeled data for SQL analytics. A data lake stores anything — raw files, JSON, images, logs — cheaply in object storage, schema-on-read, which suits ML and semi-structured data but historically lacked reliability. The lakehouse merges the two: lake-cheap storage with warehouse-grade transactions and performance via open table formats.
WAREHOUSE structured, modeled, SQL, reliable (BigQuery/Snowflake/Redshift)
great for BI; awkward for raw/unstructured
DATA LAKE any format, cheap object storage, (files on S3/GCS/ADLS)
schema-on-read; great for ML + raw;
historically NO transactions / easy to turn into a "swamp"
LAKEHOUSE lake storage + warehouse reliability (Delta, Iceberg, Hudi
(ACID, schema, time travel) via a on top of the lake)
table format -> one place for BI AND MLThe lake’s performance rests on the file format. CSV/JSON are row-oriented, untyped, and uncompressed — fine for interchange, terrible to analyze at scale. Parquet (and ORC) are columnar, typed, and compressed, so an engine reads only needed columns and skips row groups via footer statistics. Storing lake data as Parquet is one of the highest-leverage choices a data engineer makes.
# same data, two formats — analytics reads them very differently
# CSV: row-oriented, untyped, uncompressed -> full scan every query
# Parquet: columnar, typed, compressed, with per-block min/max stats
# convert once at ingestion; every downstream query benefits
import pandas as pd
df = pd.read_csv("events.csv")
df.to_parquet("events.parquet", compression="snappy")
# a query for one column now reads one column's blocks, skipping the rest,
# and gets ~5-10x compression. Land lake data as Parquet, not CSV.A pile of Parquet files has no notion of a consistent update — a reader can see a half-written batch. Open table formats (Delta Lake, Apache Iceberg, Hudi) add a transaction log over the files, giving ACID commits, schema enforcement and evolution, and time travel. This is what turns an unreliable lake into a lakehouse you can trust for production tables.
Plain Parquet files: no atomic commit -> a reader can catch a half-written
job; concurrent writes corrupt state; no schema enforcement.
TABLE FORMAT (Delta / Iceberg / Hudi) = Parquet data + a transaction LOG:
ACID commits readers never see a partial write
schema enforcement reject rows that don't match; evolve schema safely
time travel query the table "as of" an earlier version/timestamp
upserts / MERGE update + delete on the lake (needed for SCD, GDPR)
This is the "house" in lakehouse — warehouse guarantees on lake storage.A practical pattern for organizing a lakehouse is medallion (bronze/silver/gold): land raw data untouched (bronze), clean and conform it (silver), then model it into business-ready aggregates (gold). It gives every table a clear purpose and makes pipelines reproducible — you can always rebuild silver and gold from immutable bronze. This layering is exactly what the dbt and Data Engineering courses build.
BRONZE raw, append-only, exactly as ingested (replayable source of truth)
| keep everything; never edit in place
v
SILVER cleaned, typed, deduped, conformed (validated, joinable)
| quality checks applied here (see Data Quality course)
v
GOLD business-level: star schemas, aggregates, (what BI / ML consume)
metrics, marts
Because BRONZE is immutable, you can always REBUILD silver+gold from it
-> reproducible pipelines. dbt models these layers as SQL (see dbt course).