A single train/test split is noisy. Cross-validation gives a stable performance estimate, and grid or random search finds the hyperparameters that make a model shine.
Why: one split can be lucky or unlucky, so cross-validation trains and tests on k different folds and averages — a far more reliable estimate of true performance. When: use it to compare models and to tune; keep a final untouched test set for the last check. Where: use StratifiedKFold for classification so each fold keeps the class balance.
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
scores = cross_val_score(
RandomForestClassifier(random_state=42),
X_train, y_train, cv=5, scoring="f1",
)
print("per-fold F1:", scores.round(3))
print(f"mean {scores.mean():.3f} +/- {scores.std():.3f}") # stable estimateWhy: default hyperparameters are rarely best — search finds good ones automatically, evaluating each candidate with cross-validation so the choice is not overfit to one split. When: GridSearchCV for a few parameters, RandomizedSearchCV when the space is large. Where: search over a pipeline so preprocessing is re-fit inside each fold (no leakage).
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
grid = {
"n_estimators": [100, 300],
"max_depth": [None, 10, 20],
"min_samples_leaf": [1, 3],
}
search = GridSearchCV(
RandomForestClassifier(random_state=42),
grid, cv=5, scoring="f1", n_jobs=-1,
)
search.fit(X_train, y_train)
print("best params:", search.best_params_)
print("best CV F1:", round(search.best_score_, 3))Why: comparing training score to cross-validation score tells you whether the model overfits (great on train, poor on CV) or underfits (poor on both) — which points to opposite fixes. When: check this gap before tuning blindly. Where: overfitting → regularize or simplify; underfitting → add features or a stronger model.
train >> CV score OVERFITTING -> more data, regularize, simplify
train ~= CV, both low UNDERFITTING -> add features, stronger model
train ~= CV, both high GOOD -> ship it (after test-set check)
The gap between train and validation is the single best diagnostic.