Missing values are everywhere and every model rejects them. Diagnose why they’re missing, then impute with a strategy you can defend — fit on train, apply to test.
Why: how you handle a missing value depends on why it is missing and how much is gone — a column that is 90% empty is different from one with a few gaps. When: quantify missingness per column first, then decide drop vs. impute. Where: dropping rows loses data; dropping a column loses a feature — impute unless a column is almost entirely empty.
import pandas as pd
miss = df.isna().mean().sort_values(ascending=False)
print(miss) # fraction missing per column
# Drop columns that are almost entirely empty (little signal to keep):
df = df.drop(columns=miss[miss > 0.6].index)Why: imputation learns a fill value (median, mean, most-frequent) from the training data and applies it everywhere, keeping every row usable. When: median for skewed numbers, mean for symmetric, most-frequent for categoricals. Where: the imputer is fit on train only — computing the median over the full dataset is leakage.
from sklearn.impute import SimpleImputer
num_imputer = SimpleImputer(strategy="median")
X_train[num_cols] = num_imputer.fit_transform(X_train[num_cols]) # learn on train
X_test[num_cols] = num_imputer.transform(X_test[num_cols]) # apply to test
cat_imputer = SimpleImputer(strategy="most_frequent")
X_train[cat_cols] = cat_imputer.fit_transform(X_train[cat_cols])
X_test[cat_cols] = cat_imputer.transform(X_test[cat_cols])Why: sometimes the fact that a value is missing carries signal — a blank "income" field may correlate with the target — so you flag it before filling. When: add a boolean "was_missing" column when missingness is not random. Where: this preserves information that plain imputation would erase.
# Flag missingness BEFORE imputing, when the gap itself may be informative:
for col in ["income", "last_login"]:
X_train[col + "_missing"] = X_train[col].isna().astype(int)
X_test[col + "_missing"] = X_test[col].isna().astype(int)
# Then impute the original columns as usual.