The architecture behind modern AI. A Transformer stacks multi-head self-attention and feed-forward layers, processing a whole sequence in parallel — no recurrence.
Why: one attention head learns one kind of relationship; running several heads in parallel lets the model attend to different patterns at once (syntax, meaning, position) and combine them. When: multi-head attention is the workhorse layer of every Transformer. Where: each head works in a lower-dimensional subspace, then the heads are concatenated and mixed.
Single head: one view of "what relates to what"
Multi-head: H heads in parallel, each a different view, concatenated
One head might track subject-verb agreement, another long-range
reference, another local phrasing. Together they're expressive.Why: the Transformer block is a fixed, repeatable unit — self-attention then a feed-forward network, each wrapped in a residual connection and layer norm — and stacking it is what builds deep models like BERT and GPT. When: use nn.TransformerEncoderLayer rather than assembling it by hand. Where: residual connections and layer norm are what let you stack many blocks without training breaking.
import torch, torch.nn as nn
layer = nn.TransformerEncoderLayer(
d_model=64, nhead=8, # embedding size, number of heads
dim_feedforward=256, batch_first=True,
)
encoder = nn.TransformerEncoder(layer, num_layers=6) # stack 6 blocks
x = torch.randn(4, 20, 64) # batch=4, seq=20, embedding=64
print(encoder(x).shape) # [4, 20, 64] — contextualized embeddingsWhy: attention has no built-in sense of order — it sees a set, not a sequence — so Transformers add positional encodings to tell the model where each token sits. When: you must add position information before the encoder, or word order is lost. Where: this is the trade for parallelism — RNNs get order for free from recurrence; Transformers inject it explicitly.
import torch
# Positional encodings are added to token embeddings so the model knows order.
seq_len, d_model = 20, 64
position = torch.arange(seq_len).unsqueeze(1)
div = torch.exp(torch.arange(0, d_model, 2) * (-9.21 / d_model))
pe = torch.zeros(seq_len, d_model)
pe[:, 0::2] = torch.sin(position * div)
pe[:, 1::2] = torch.cos(position * div)
# embeddings = token_embeddings + pe -> now order-aware