Doing these steps by hand invites leakage and bugs. A scikit-learn Pipeline chains preprocessing and the model into one object that fits on train and applies everywhere.
Why: a Pipeline bundles every transform plus the model into one estimator, so fit learns all parameters from train and predict applies them — making leakage structurally impossible and cross-validation correct. When: use a pipeline for every real project; never hand-chain transforms. Where: the whole pipeline serializes as one artifact, so training and serving use identical preprocessing.
WITHOUT a pipeline: fit scaler, transform train, transform test, fit
model, predict... easy to leak, easy to forget a step at serving time.
WITH a pipeline: pipe.fit(X_train, y_train) -> learns EVERYTHING on train
pipe.predict(X_test) -> applies EVERYTHING
one object to save, load, and serve. No drift.Why: real tables mix numeric and categorical columns that need different preprocessing — ColumnTransformer routes each column group to the right transformer in parallel. When: use it whenever you have both kinds of features (almost always). Where: it outputs one combined matrix ready for the model.
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
num_cols = ["age", "income"]
cat_cols = ["city", "role"]
num = Pipeline([("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler())])
cat = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
("onehot", OneHotEncoder(handle_unknown="ignore"))])
pre = ColumnTransformer([("num", num, num_cols), ("cat", cat, cat_cols)])Why: attaching the model as the pipeline’s final step means one fit call trains the whole thing on train data only, and one predict call runs the full transform-then-predict path — no step can leak or be forgotten. When: this is the shape every project should ship. Where: cross-validation on this pipeline re-fits preprocessing inside each fold, giving honest scores.
from sklearn.linear_model import LogisticRegression
model = Pipeline([
("pre", pre), # all preprocessing from above
("clf", LogisticRegression(max_iter=1000)),
])
model.fit(X_train, y_train) # learns imputers, scaler, encoder, model
print("test accuracy:", model.score(X_test, y_test))
import joblib
joblib.dump(model, "model.joblib") # one artifact — preprocessing included