What is a transformer?
A transformer is the assembly. Take embeddings, attention, and a small neural network, wire them into a repeatable block, and stack that block dozens of times. That stack is what an LLM is.
01The answer, then the intuition
The parts you already know, wired together
You've met every piece. A transformer just connects them. Text becomes tokens, tokens become embeddings, and then the data flows through a stack of identical blocks. Inside each block, attention lets tokens share information, and a small neural network processes each one. Repeat, and out the top come probabilities for the next token.
Click through the architecture — each stage is one of the earlier chapters doing its job:
The transformer — click any stage
Data flows up. The shaded block is the unit that repeats.
Tap any stage in the diagram to see what it does — and which chapter it comes from.
02Mechanics
One block, repeated
The whole architecture is one block applied over and over. Inside it, two ideas you've seen, plus two pieces of plumbing that make deep stacks trainable:
- Self-attention mixes information across tokens — the only place they talk to each other.
- The feed-forward network then processes each token independently. This is where most of a model's parameters actually live.
- Residual connections add a layer's input back to its output, so information (and gradients) can skip straight through a hundred layers without vanishing.
- Layer normalization rescales the numbers at each step to keep training stable.
One detail matters: attention is order-blind — shuffle the tokens and it gives the same answer. So before the first block, each embedding gets a positional encoding added, a signal telling the model where each token sits. Stack ~12 blocks for a small model, ~100 for a frontier one, cap it with an output layer that turns the final vectors into next-token probabilities, and you have GPT.
04The math
expand ▾The block as two residual steps
04The math
expand ▾The block as two residual steps
Let $\mathbf{x}$ be the matrix of token vectors entering a block. The block is exactly two residual-and-normalize steps — first attention, then the feed-forward network:
where the feed-forward network is a two-layer MLP applied to each token, $\text{FFN}(\mathbf{x}) = \max(0,\,\mathbf{x}W_1)\,W_2$. Crucially the block maps $\mathbb{R}^{n\times d} \to \mathbb{R}^{n\times d}$ — same shape in, same shape out — which is precisely why you can stack it $N$ times:
The final vectors are projected by an unembedding matrix to vocabulary-sized scores (logits) and softmaxed into the probability of the next token. That's the entire forward pass of a GPT.
05The code
expand ▾A transformer block in numpy
05The code
expand ▾A transformer block in numpy
The block, composing the pieces from the earlier chapters. Note the output shape equals the input shape — that's the stacking property.
block.py
import numpy as np
np.random.seed(0)
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True)); return e / e.sum(axis=-1, keepdims=True)
def attention(x, Wq, Wk, Wv):
Q, K, V = x@Wq, x@Wk, x@Wv
return softmax(Q@K.T / np.sqrt(Q.shape[-1])) @ V
def layernorm(x):
return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + 1e-5)
def ffn(x, W1, W2):
return np.maximum(0, x@W1) @ W2 # 2-layer ReLU MLP
def block(x, Wq, Wk, Wv, W1, W2):
x = layernorm(x + attention(x, Wq, Wk, Wv)) # attention + residual + norm
x = layernorm(x + ffn(x, W1, W2)) # feed-forward + residual + norm
return x
x = np.random.randn(3, 4) # 3 tokens, width 4
Wq, Wk, Wv = (np.random.randn(4, 4) for _ in range(3))
W1, W2 = np.random.randn(4, 8), np.random.randn(8, 4) # FFN hidden = 8
print(block(x, Wq, Wk, Wv, W1, W2).shape) # (3, 4) — stack it as many times as you like07Going deeper
expand ▾The primary sources
07Going deeper
expand ▾The primary sources
Vaswani et al. (2017) — Attention Is All You Need · the transformer architecture itself.
Radford et al. (2018) — GPT · the decoder-only transformer that became the LLM.
Phuong & Hutter (2022) — Formal Algorithms for Transformers · the architecture written out precisely.
Alammar — The Illustrated Transformer · the canonical visual walkthrough.
"First Principles — how AI actually works", chapter "What is a transformer?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.