More features is not better — noise, redundancy, and leakage hide among them. Drop the features that don’t help so your model is simpler, faster, and more robust.
Why: irrelevant and redundant features add noise, slow training, and increase overfitting — cutting them often improves the model and always makes it simpler to reason about. When: select features after engineering, before final training. Where: selection also helps you catch leaky features (ones suspiciously predictive of the target).
Three families of feature selection:
FILTER score each feature vs. target, cheap (variance, correlation)
WRAPPER train models on feature subsets (RFE) — slower, better
EMBEDDED the model selects during training (Lasso, tree importances)
Start with filters to prune obvious junk, then use embedded importances.Why: a feature that never changes (zero variance) carries no information, and a quick univariate test ranks how strongly each feature relates to the target — both are fast first passes. When: drop constant columns, then keep the top-k by univariate score. Where: filter methods ignore feature interactions, so treat them as a coarse first cut.
from sklearn.feature_selection import VarianceThreshold, SelectKBest, f_classif
# Drop near-constant columns:
vt = VarianceThreshold(threshold=0.0)
X_train_v = vt.fit_transform(X_train)
# Keep the k features most related to the target (fit on train!):
skb = SelectKBest(score_func=f_classif, k=10)
X_train_k = skb.fit_transform(X_train, y_train)
X_test_k = skb.transform(X_test)Why: models like Lasso and tree ensembles score feature usefulness as a side effect of training, giving you a principled ranking that accounts for interactions. When: use tree importances or Lasso coefficients to drop low-signal features. Where: Lasso drives useless coefficients to exactly zero, doubling as a selector (covered in the scikit-learn course).
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X_train, y_train)
importances = pd.Series(rf.feature_importances_, index=X_train.columns)
print(importances.sort_values(ascending=False).head(10))
# Keep features above a small importance threshold.
keep = importances[importances > 0.01].index
X_train_sel, X_test_sel = X_train[keep], X_test[keep]