When you have hundreds of correlated features, PCA compresses them into a handful that keep most of the information — fighting the curse of dimensionality.
Why: as features grow, data becomes sparse and distances lose meaning, so models need exponentially more data — reducing dimensions restores signal density. When: use it with many correlated features (images, sensors, text vectors), especially before distance-based models. Where: unlike feature selection (which drops columns), PCA creates new combined features.
SELECTION keeps a subset of the ORIGINAL features (interpretable)
REDUCTION builds NEW features that combine the originals (compact)
PCA finds the directions of greatest variance and projects onto the top
few — keeping most of the information in far fewer dimensions.Why: PCA finds the axes along which the data varies most and keeps only the top few, compressing many correlated features into a small set that retains most of the variance. When: use it to speed up training, denoise, or fight the curse of dimensionality. Where: PCA needs scaled input (it is variance-based), and it must be fit on train only.
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
X_train_s = StandardScaler().fit_transform(X_train) # PCA needs scaling
pca = PCA(n_components=0.95) # keep enough components for 95% variance
X_train_p = pca.fit_transform(X_train_s)
print("kept", pca.n_components_, "components")
print(pca.explained_variance_ratio_.cumsum()) # variance retainedWhy: projecting high-dimensional data down to two components lets you SEE structure — clusters, separation between classes — that is invisible in a table. When: use PCA (or t-SNE for non-linear structure) to plot data before modeling. Where: 2-component PCA is for insight; keep more components when reducing for a model.
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
coords = PCA(n_components=2).fit_transform(X_train_s)
plt.scatter(coords[:, 0], coords[:, 1], c=y_train, cmap="viridis", s=10)
plt.xlabel("PC1"); plt.ylabel("PC2"); plt.title("Data in 2D")
plt.show() # do classes separate? clusters appear? -> informs modeling