Text isn’t numbers. Turn it into tokens, map tokens to learnable embedding vectors, and understand the preprocessing pipeline behind every language model.
Why: models cannot read strings, so text is split into tokens (words or subword pieces) and each token is mapped to an integer id — the first step of every NLP pipeline. When: modern models use subword tokenization (BPE, WordPiece) so any word, even unseen ones, splits into known pieces. Where: classic steps like stemming and lemmatization normalize words but are mostly replaced by subword tokenization today.
"unbelievable" --tokenize--> ["un", "believ", "able"] --ids--> [45, 1203, 88]
CLASSIC NLP (still useful for feature-based models):
tokenization split into words
stemming "running" -> "run" (crude chop)
lemmatization "better" -> "good" (dictionary form)
MODERN: subword tokenizers handle unknown words by splitting them.Why: a token id is just a label, so it is mapped to a dense embedding vector that the model learns — similar words end up with similar vectors, giving the model a sense of meaning. When: an nn.Embedding layer is the first layer of nearly every text model. Where: the embedding matrix is (vocab_size, embedding_dim) and is learned during training like any other weights.
import torch, torch.nn as nn
vocab_size, embed_dim = 10000, 64
embedding = nn.Embedding(vocab_size, embed_dim) # learnable lookup table
token_ids = torch.tensor([[45, 1203, 88, 0]]) # one tokenized sentence
vectors = embedding(token_ids)
print(vectors.shape) # [1, 4, 64] — each token is now a 64-d vector
# These vectors feed into an RNN or a Transformer.Why: composing an embedding layer with a sequence encoder and a head gives a complete text classifier — the pattern behind sentiment analysis, spam detection, and topic tagging. When: embed the token ids, encode the sequence, pool to one vector, then classify. Where: in practice you start from a pretrained tokenizer and embeddings rather than training them from scratch (next lesson).
import torch.nn as nn
class TextClassifier(nn.Module):
def __init__(self, vocab, embed, hidden, classes):
super().__init__()
self.embed = nn.Embedding(vocab, embed)
self.rnn = nn.LSTM(embed, hidden, batch_first=True)
self.head = nn.Linear(hidden, classes)
def forward(self, ids):
vecs = self.embed(ids) # ids -> vectors
_, (h, _) = self.rnn(vecs) # encode the sequence
return self.head(h[-1]) # classify the summary