The T in GPT stands for Transformer. The architecture traces back to a single paper posted to arXiv in June 2017: “Attention Is All You Need,” written by eight researchers, most of them at Google Brain and Google Research.
You’ve probably seen its architecture diagram in a dozen explainers. Here we use the paper’s own figures. The paper carries a note from Google granting permission to reproduce its tables and figures in journalistic or scholarly work with attribution, so you can follow along with the originals instead of redrawn versions.
The problem: reading one word at a time
Before Transformers, tasks like machine translation were dominated by recurrent neural networks (RNNs), especially LSTMs. They read a sentence left to right, one word at a time, and each step depends on the result of the previous one.
The paper calls this out directly. Because the computation is sequential, you can’t parallelize within a training example, and that gets worse as sentences get longer. The authors’ answer was to drop recurrence entirely and model the relationships between words using attention alone. Hence the title.
Figure 1: one side reads, the other writes

Image: Vaswani et al., “Attention Is All You Need,” Figure 1 (reproduced with Google’s permission)
The diagram splits into two halves.
- Left, the encoder. It reads the input sentence (say, the English text to translate) and turns each word into a vector that carries its meaning in context. The base model stacks six identical layers (N=6), each with a multi-head attention block and a feed-forward network.
- Right, the decoder. It produces the output sentence (say, the German translation) one word at a time. It also has six layers, with one extra block in each: a middle attention layer that looks at the encoder’s output to decide which part of the source sentence matters for the next word.
Notice the word “Masked” on the decoder’s bottom attention block. While generating, the model must not peek at words it hasn’t written yet, so future positions are hidden. Keep that in mind; it comes back when we get to GPT.
Also notice positional encoding at the bottom. With recurrence gone, the model has no built-in sense of word order, so each word gets a position signal added to it, built from sine and cosine functions.
Figure 2: attention decides which words to look at, and how much

Image: Vaswani et al., “Attention Is All You Need,” Figure 2 (reproduced with Google’s permission)
The left diagram is a single attention step. Q, K, and V stand for Query, Key, and Value. A library is a handy analogy:
- The word being processed asks a query: which other words are relevant to me?
- Every word in the sentence offers a key. Multiplying query by key (MatMul) gives a match score.
- Softmax turns those scores into weights that add up to 1.
- The word’s new representation is a weighted mix of every word’s value, so strongly related words contribute the most.
The paper’s formula fits on one line: softmax(QKᵀ / √dₖ) V. It also explains the “Scale” step. When the vector size dₖ is large, the dot products grow large, pushing softmax toward extreme outputs where gradients become tiny. Dividing by √dₖ keeps training stable.
The right diagram, multi-head attention, runs several of these in parallel. The base model uses eight heads (h=8), each looking at a 64-dimensional slice of the 512-dimensional representation. Instead of one way of looking at a sentence, the model gets eight.
Figure 3: who does “its” refer to?

Image: Vaswani et al., “Attention Is All You Need,” part of Figure 4 (reproduced with Google’s permission)
This figure shows what attention actually learns. In “The Law will never be perfect, but its application should be just,” two heads in layer 5 of 6 connect “its” strongly to “Law” and “application.” The model has picked up, on its own, that “its” refers to the law.
The paper is careful here, describing these heads as “apparently involved” in resolving the reference. A few visualizations don’t explain everything happening inside a model, and it’s worth reading them with that caution.
The results: faster training, better translations
The paper made its case with translation benchmarks.
| Base model | Big model | |
|---|---|---|
| Parameters | ~65 million | ~213 million |
| Training time (8 × NVIDIA P100) | 12 hours | 3.5 days |
| English→German BLEU | 27.3 | 28.4 |
| English→French BLEU | 38.1 | 41.8 |
The big model’s 28.4 on English→German beat the best previously reported results, ensembles included, by more than 2 BLEU. Even the base model topped every previously published model. The real story was cost: eight GPUs for 12 hours, or 3.5 days at most.
Reading closely turns up one small inconsistency. In the current arXiv version (v7), the English→French score is 41.8 in the abstract and results table but 41.0 in the paragraph discussing the results. Most citations use 41.8, but it’s worth knowing the discrepancy exists if you quote the number.
GPT kept the right side, BERT kept the left
With Figure 1 in mind, the difference between the two best-known descendants becomes easy to see.
GPT took the decoder. OpenAI’s first GPT paper, “Improving Language Understanding by Generative Pre-Training,” says it uses a multi-layer Transformer decoder as its language model: specifically, a 12-layer decoder-only model with masked self-attention. With no encoder, the middle block on the right side of Figure 1 disappears, leaving masked attention and feed-forward layers. The model predicts the next word from the words before it, which is exactly what text generation needs.
BERT took the encoder. The name says so: Bidirectional Encoder Representations from Transformers. Without the mask, it looks at words on both sides at once, which makes it strong at understanding and classifying text rather than writing it. It was posted to arXiv in October 2018.
| Original Transformer | GPT | BERT | |
|---|---|---|---|
| Uses | Encoder + decoder | Decoder | Encoder |
| Direction | Encoder both ways, decoder left-to-right | Left-to-right | Both ways |
| Good at | Translation | Generation | Understanding, classification |
Why this diagram still matters
Once you know these three figures, AI news gets easier to parse. A model’s “context window” is how many tokens attention can look at in one pass. Because attention scores every pair of words, doubling the length means four times as many scores; the paper’s complexity table lists self-attention as growing with the square of sequence length. That’s why long context is expensive. “Parameter count” is the total number of learned values inside the layers in Figure 1.
As GPT and BERT show, later models largely evolved by taking parts of this diagram and scaling or reshaping them. The next time a new model is announced, asking “which part of Figure 1 did they change?” is a surprisingly good way into the story.