Two choices shape a network: the activation that gives it non-linearity, and the loss that defines what "wrong" means. Pick the right ones for your task.
Why: without a non-linear activation between layers, a deep network is mathematically just one linear layer — activations are what let it learn curves and complex patterns. When: ReLU is the default for hidden layers; sigmoid and softmax appear at the output to produce probabilities. Where: the output activation is dictated by the task, not preference.
ACTIVATION USE RANGE
ReLU hidden layers (default) [0, inf)
Sigmoid binary output -> probability (0, 1)
Softmax multi-class output -> probs sums to 1
Tanh hidden layers, zero-centered (-1, 1)
Hidden layers: ReLU. Output layer: match the task (see below).Why: the loss defines what the network optimizes, so it must match the problem — regression uses MSE, binary classification uses binary cross-entropy, multi-class uses cross-entropy. When: pick the loss first; it determines the output layer too. Where: PyTorch’s CrossEntropyLoss expects raw scores (logits) and applies softmax internally — do not add a softmax yourself.
TASK LOSS OUTPUT LAYER
regression nn.MSELoss linear (no activation)
binary classify nn.BCEWithLogitsLoss linear (1 unit, logit)
multi-class classify nn.CrossEntropyLoss linear (C units, logits)
Note: the *WithLogits / CrossEntropy losses apply sigmoid/softmax
INTERNALLY. Feed them raw logits, not probabilities.Why: seeing the loss as a single number that a criterion computes from predictions and targets makes the training loop concrete — this number is what backward() differentiates. When: instantiate the criterion once, call it every step. Where: for CrossEntropyLoss, targets are integer class indices, not one-hot vectors.
import torch, torch.nn as nn
logits = torch.tensor([[2.0, 0.5, 0.1], # raw scores for 3 classes
[0.1, 0.2, 3.0]])
targets = torch.tensor([0, 2]) # correct class INDEX per sample
criterion = nn.CrossEntropyLoss() # applies softmax internally
loss = criterion(logits, targets)
print(loss.item()) # one number to minimize