Where does all the data live? A lake stores raw files cheaply; a warehouse stores structured tables for fast queries; a lakehouse blends both. Plus the file format that matters.
Why: the two classic stores serve different needs — a data lake holds raw files of any shape cheaply (great for ML’s varied inputs), a warehouse holds clean structured tables optimized for SQL analytics — and the lakehouse combines cheap storage with warehouse features. When: land raw data in a lake, curate structured tables in a warehouse (or a lakehouse table format). Where: ML pipelines usually read raw from the lake and write features to structured tables.
DATA LAKE raw files (any format) on cheap object storage (S3/GCS)
schema-on-read, stores everything, ML-friendly
DATA WAREHOUSE structured tables, schema-on-write, fast SQL analytics
BigQuery, Snowflake, Redshift
LAKEHOUSE lake storage + warehouse features (ACID, schema)
Delta Lake, Apache Iceberg, Hudi
Typical ML flow: land raw in the LAKE -> curate features in a TABLE.Why: how data is stored on disk hugely affects speed and cost — Parquet is columnar and compressed, so a query reading two columns of a hundred touches only those two, unlike row-based CSV which reads everything. When: store analytical and ML data as Parquet, not CSV, once it is beyond toy size. Where: columnar layout is why warehouses and Spark are fast; it also preserves data types that CSV loses.
CSV (row-based) read 2 of 100 columns -> still scans every byte
PARQUET (columnar) read 2 of 100 columns -> reads only those 2
Parquet: columnar + compressed + keeps dtypes + splittable.
Rule: humans get CSV; pipelines and ML get Parquet.Why: even Parquet is slow if a query scans a whole multi-terabyte dataset — partitioning splits data into folders by a column (usually date) so a query for one day reads only that folder. When: partition large tables by the column you filter on most, typically the event date. Where: "partition pruning" is what lets queries stay fast as data grows; over-partitioning (tiny files) hurts, so choose a sensible granularity.
# Partitioned layout on the lake — one folder per day:
s3://lake/events/
dt=2024-07-04/part-0000.parquet
dt=2024-07-05/part-0000.parquet
dt=2024-07-06/part-0000.parquet
# A query for one day reads ONLY that folder (partition pruning):
# SELECT * FROM events WHERE dt = '2024-07-06'
# Partition by what you filter on (usually date). Avoid thousands of tiny files.