Training is just following gradients downhill. PyTorch computes them for you — set requires_grad, call backward, and read the gradient. This is the engine of learning.
Why: a network learns by measuring how wrong it is (the loss), computing the gradient of that loss with respect to every weight, and nudging each weight in the direction that lowers the loss — repeated thousands of times. When: this loop underlies every model in this course. Where: autograd is the part that computes those gradients automatically, so you never derive them by hand.
forward: inputs -> model -> prediction -> loss (how wrong?)
|
backward: gradient of loss w.r.t. every weight <- (autograd)
|
update: weight -= learning_rate * gradient (step downhill)
repeat until the loss stops falling.Why: marking a tensor with requires_grad tells PyTorch to record every operation on it, so calling backward() on the result computes the gradient with respect to that tensor. When: model parameters have requires_grad=True automatically; you rarely set it by hand. Where: the gradient lands in the tensor’s .grad attribute.
import torch
w = torch.tensor([2.0], requires_grad=True) # track operations on w
loss = (w - 5) ** 2 # a toy loss, minimized at w = 5
loss.backward() # compute d(loss)/d(w)
print(w.grad) # tensor([-6.]) -> loss decreases as w increases
# One gradient-descent step by hand:
with torch.no_grad():
w -= 0.1 * w.grad # move w toward 5Why: running the measure-gradient-step loop by hand makes the abstraction concrete — you literally watch a parameter slide to the value that minimizes the loss. When: this is exactly what an optimizer automates in the next lessons. Where: you must zero the gradient each step, because PyTorch accumulates gradients by default.
import torch
w = torch.tensor([0.0], requires_grad=True)
lr = 0.1
for step in range(20):
loss = (w - 5) ** 2
loss.backward() # fills w.grad
with torch.no_grad():
w -= lr * w.grad # step downhill
w.grad.zero_() # MUST reset — grads accumulate otherwise
print("w converged to", w.item()) # ~5.0