You rarely train from scratch. Start from a model pretrained on millions of examples, adapt it to your task with little data, and get strong results fast.
Why: models pretrained on huge datasets have already learned general features — edges and textures for vision, grammar and meaning for language — so adapting one to your task needs far less data and compute than training from zero. When: this is the default for almost every real project; from-scratch training is the exception. Where: you reuse the learned feature extractor and only retrain a small task-specific head, or lightly fine-tune.
FROM SCRATCH: millions of examples, days of compute, easy to overfit
TRANSFER: start pretrained -> adapt with hundreds/thousands of
examples -> strong results in minutes to hours
Two strategies:
FEATURE EXTRACTION freeze the backbone, train only a new head
FINE-TUNING unfreeze some/all layers, train at a low LRWhy: freezing a pretrained backbone and training only a new final layer reuses all its learned visual features while learning just your classes — fast, data-efficient, and hard to overfit. When: use it when your dataset is small or similar to the pretraining data. Where: set requires_grad=False on the backbone so only the new head updates.
import torch, torch.nn as nn
from torchvision import models
model = models.resnet18(weights="IMAGENET1K_V1") # pretrained backbone
for p in model.parameters(): # freeze everything
p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 3) # new head: 3 of YOUR classes
# Only model.fc trains. Same training loop as the PyTorch course.
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)Why: the same idea powers NLP — Hugging Face gives you pretrained tokenizers and Transformer models you fine-tune on your task in a few lines, which is how most text classifiers are built today. When: reach for a pretrained model instead of training embeddings and a Transformer from scratch. Where: the tokenizer must match the model checkpoint, and fine-tuning uses a low learning rate (see the fine-tuning project checklist on the roadmap).
# pip install transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification
name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(name) # matching tokenizer
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)
inputs = tokenizer(["I loved this film"], return_tensors="pt",
padding=True, truncation=True)
logits = model(**inputs).logits # fine-tune on your labeled data with a low LR