RNNs process sequences step by step and forget. Attention lets a model look at every position at once and weigh what matters — the idea that made Transformers possible.
Why: an RNN squeezes a whole sequence into one hidden state and struggles with long-range links; attention instead lets each position directly look at every other position and decide how much to focus on each — no information bottleneck. When: attention shines when relationships span long distances (a pronoun and its noun paragraphs apart). Where: it replaced recurrence as the dominant way to model sequences.
RNN: compress everything into one hidden state, step by step
-> long-range dependencies fade
ATTENTION: for each position, compute a weighted sum over ALL
positions, where the weights say "how relevant is each
other token to me?"
No sequential bottleneck; distant tokens are one step away.Why: attention works by comparing a query (what I am looking for) against every key (what each position offers) to get relevance weights, then taking a weighted sum of the values — the actual content. When: this Q/K/V mechanism is the atom of every Transformer. Where: the softmax turns raw relevance scores into weights that sum to one.
import torch, torch.nn.functional as F
# One attention head, by hand, to see the mechanism.
seq_len, dim = 4, 8
Q = torch.randn(seq_len, dim) # queries
K = torch.randn(seq_len, dim) # keys
V = torch.randn(seq_len, dim) # values
scores = Q @ K.T / dim**0.5 # relevance of every position to every other
weights = F.softmax(scores, dim=-1) # normalize to sum to 1
output = weights @ V # weighted sum of values
print(output.shape) # [4, 8] — each position, re-mixed by relevanceWhy: when the queries, keys, and values all come from the same sequence, the mechanism is self-attention — every token rewrites itself as a blend of all tokens, capturing context. When: this is the core operation inside a Transformer encoder. Where: PyTorch provides it directly, so you rarely hand-roll it in production.
import torch, torch.nn as nn
# PyTorch's built-in multi-head self-attention.
x = torch.randn(4, 10, 32) # batch=4, seq=10, dim=32
attn = nn.MultiheadAttention(embed_dim=32, num_heads=4, batch_first=True)
# Self-attention: query = key = value = x
out, weights = attn(x, x, x)
print(out.shape) # [4, 10, 32] — each token re-mixed by context