Sequences — text, time series, audio — have order that matters. Recurrent networks process them step by step, carrying a memory; LSTM and GRU fix their short memory.
Why: a CNN or MLP sees a fixed input all at once, but sequences unfold over time — an RNN processes one element at a time while carrying a hidden state that summarizes everything seen so far. When: use recurrent models for ordered data where earlier elements inform later ones. Where: the hidden state IS the memory passed from step to step.
x1 -> [RNN] -> h1
|
x2 -> [RNN] -> h2 each step takes the input AND the previous
| hidden state -> the network "remembers"
x3 -> [RNN] -> h3
|
... h_t summarizes x1..x_tWhy: plain RNNs forget quickly — gradients vanish over long sequences, so they cannot connect distant events — LSTM and GRU add gates that decide what to keep, remember, and forget, enabling long-range memory. When: default to LSTM or GRU over a vanilla RNN for anything but the shortest sequences. Where: GRU is lighter with fewer parameters; LSTM is slightly more expressive.
import torch, torch.nn as nn
x = torch.randn(4, 20, 10) # batch=4, sequence length=20, features=10
lstm = nn.LSTM(input_size=10, hidden_size=32, batch_first=True)
output, (h_n, c_n) = lstm(x)
print(output.shape) # [4, 20, 32] — hidden state at every step
print(h_n.shape) # [1, 4, 32] — final hidden state
gru = nn.GRU(input_size=10, hidden_size=32, batch_first=True) # lighter optionWhy: to classify a whole sequence (sentiment of a sentence, type of a signal) you run it through the recurrent layer and feed the FINAL hidden state — the summary of the whole sequence — to a dense head. When: use the last hidden state for sequence-level tasks; use all outputs for per-step tasks like tagging. Where: this is the template for text or time-series classification with RNNs.
import torch.nn as nn
class SeqClassifier(nn.Module):
def __init__(self, in_size, hidden, num_classes):
super().__init__()
self.rnn = nn.LSTM(in_size, hidden, batch_first=True)
self.head = nn.Linear(hidden, num_classes)
def forward(self, x):
_, (h_n, _) = self.rnn(x) # h_n: final hidden state
return self.head(h_n[-1]) # summarize -> classify
model = SeqClassifier(in_size=10, hidden=32, num_classes=2)