Accuracy lies on imbalanced data. Use precision, recall, F1, the confusion matrix, and ROC-AUC to know what your classifier is really doing — and where it fails.
Why: on imbalanced data (99% one class) a model that always predicts the majority scores 99% accuracy while being useless — you need metrics that expose the errors that matter. When: always look past accuracy for fraud, disease, churn, or any rare-positive problem. Where: precision and recall separate the two kinds of mistake accuracy blends together.
Confusion matrix (binary):
predicted 0 predicted 1
actual 0 TN FP (false alarm)
actual 1 FN TP (missed positive)
precision = TP / (TP + FP) "when I say positive, am I right?"
recall = TP / (TP + FN) "of all positives, how many did I catch?"
F1 = harmonic mean of precision and recallWhy: precision, recall, and F1 per class — plus the confusion matrix — tell you exactly where the model fails, not just how often. When: read them together; a high-precision, low-recall model is very different from the reverse, and which you want depends on the cost of each error. Where: one function prints all of it.
from sklearn.metrics import classification_report, confusion_matrix
preds = clf.predict(X_test)
print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds))
# precision/recall/f1 per class + support. Look at the MINORITY class row.Why: a classifier outputs probabilities, and the 0.5 cutoff is arbitrary — ROC-AUC measures ranking quality across all thresholds, and you can move the threshold to trade precision for recall. When: use AUC to compare models independent of threshold; tune the threshold to the business cost of each error. Where: AUC needs probabilities (predict_proba), not hard labels.
from sklearn.metrics import roc_auc_score, log_loss
probs = clf.predict_proba(X_test)[:, 1]
print("ROC-AUC:", roc_auc_score(y_test, probs)) # 0.5 = random, 1.0 = perfect
print("log loss:", log_loss(y_test, probs)) # penalizes confident mistakes
# Move the threshold to favor recall (catch more positives):
custom_preds = (probs >= 0.3).astype(int)