If attention is an engine, the Transformer is the whole car built around it. Introduced in the 2017 paper whose title says it all, “Attention Is All You Need” (Vaswani et al., 2017), it threw away the sequential recurrence that machine translation had relied on and replaced it with stacks of attention, processed all at once. That single architectural bet powers essentially every large language model you have heard of, from BERT to the GPT family. This post assembles the Transformer piece by piece, keeping each part ELI5-simple while writing down the mathematics that makes it tick. If you have not met attention yet, read the companion post first, everything here is built on it.

1. The one-sentence idea

A Transformer takes a sequence of words, and repeatedly lets every word look at every other word (attention) and then think for itself (a small neural network), over and over, until the representations are rich enough to do the task, translate, summarise, or predict the next word.

That is genuinely most of it. The rest of this post is about the plumbing that makes this idea trainable : how to tell the model the order of the words, how to keep gradients healthy across a deep stack, and how the two halves (encoder and decoder) cooperate.

2. Words in, numbers in : embeddings and order

A model cannot chew on letters, so each token (word or sub-word) is first mapped to a vector, its embedding. A sentence of $n$ tokens becomes a matrix $X \in \Bbb{R}^{n \times d}$ of $n$ rows, each of dimension $d$.

But attention has a curious blind spot : it treats its input as a bag of words. Shuffle the sentence and the attention maths gives the same answer, because a weighted average does not care about order. “Dog bites man” and “man bites dog” would look identical, which is clearly unacceptable. The fix is to add a position signal to every embedding.

The original Transformer uses fixed sinusoidal positional encodings : for position $pos$ and dimension $i$,

\[\begin{equation} \begin{aligned} PE(pos, 2i) &= \sin\!\left( \frac{pos}{10000^{\,2i/d}} \right), \\ PE(pos, 2i+1) &= \cos\!\left( \frac{pos}{10000^{\,2i/d}} \right). \end{aligned} \label{eq:posenc} \end{equation}\]

In plain words : each position gets a unique “fingerprint” made of sine waves of many different wavelengths, fast wiggles for fine position, slow waves for coarse position, just like the hour, minute and second hands of a clock together pin down a unique time. Because sines and cosines shift predictably, the model can also learn relative offsets (“three words later”) easily.

Figure 1 - Sinusoidal positional encodings $\eqref{eq:posenc}$. Different embedding dimensions oscillate at different wavelengths ; reading one vertical slice gives a position its unique code. The scanning line sweeps through positions.

3. The repeating block

The heart of the Transformer is one block, stacked $N$ times (the paper uses $N = 6$) ¹. A block does two things in sequence : mix words with multi-head self-attention, then process each word with a small feed-forward network. Two extra tricks keep such a deep stack trainable.

• Residual connections. Each sub-layer computes $x + \operatorname{SubLayer}(x)$, not just $\operatorname{SubLayer}(x)$. The input is added back, so information (and gradient) can always flow straight through. ELI5 : the block only has to learn the “edit”, not re-copy everything it received.
• Layer normalisation. After each add, the values are renormalised to a stable scale, which keeps training smooth.

Putting it together, one encoder block computes

\[\begin{equation} \begin{aligned} Z &= \operatorname{LayerNorm}\big(X + \operatorname{MultiHead}(X, X, X)\big), \\ \operatorname{Block}(X) &= \operatorname{LayerNorm}\big(Z + \operatorname{FFN}(Z)\big), \end{aligned} \label{eq:block} \end{equation}\]

where the position-wise feed-forward network is just two linear layers with a non-linearity, applied to each word identically :

\[\begin{equation} \operatorname{FFN}(z) = \max(0,\; z W_1 + b_1)\, W_2 + b_2. \label{eq:ffn} \end{equation}\]

Figure 2 - One Transformer block $\eqref{eq:block}$. Data flows bottom to top (blue) : self-attention, then a feed-forward network, each wrapped in a residual "skip" (orange) and layer normalisation. Stack this block $N$ times to get an encoder.

4. Encoder and decoder

The full translation model has two towers (Figure 3). ELI5 : the encoder reads the source sentence and builds an understanding of it ; the decoder writes the translation one word at a time, consulting that understanding.

The decoder block has one extra ingredient and one twist :

• Cross-attention. A middle attention layer where the queries come from the translation-so-far but the keys and values come from the encoder’s output. This is how the decoder “looks back” at the source, exactly the alignment that attention was originally invented for (Bahdanau et al., 2015).
• Masked self-attention. When writing word $t$, the decoder must not peek at words $t+1, t+2, \dots$ (they do not exist yet at generation time). We enforce this by setting the forbidden scores to $-\infty$ before the softmax, so their weight becomes $0$ :

\[\begin{equation} \big(\text{mask}\big)_{ij} = \begin{cases} 0 & j \le i \quad (\text{allowed : look at past and present}) \\ -\infty & j > i \quad (\text{forbidden : the future}) \end{cases} \label{eq:mask} \end{equation}\]

Figure 3 - The encoder–decoder Transformer. The encoder (blue) turns the source into a set of key/value vectors ; the decoder (green) generates the target word by word, using masked self-attention over what it has written and cross-attention (orange) into the encoder's output.

5. Generating text, one word at a time

At the top of the decoder, a linear layer plus a softmax turns the final vector into a probability distribution over the whole vocabulary :

\[\begin{equation} p(\text{next word}) = \operatorname{softmax}(h\, W_{\text{vocab}} + b). \label{eq:head} \end{equation}\]

The model picks a word, appends it to the sentence, and runs again to produce the next, this is autoregressive generation. ELI5 : it writes like you texting with predictive keyboard, one suggestion at a time, each new word feeding back in as context. The masking $\eqref{eq:mask}$ is what makes training match this behaviour : during training the model sees the whole target sentence but is forbidden from looking ahead, so it learns to predict each word from only the words before it.

6. Why Transformers took over

• Full parallelism. Unlike recurrent models that must process word 1 before word 2, a Transformer handles all positions simultaneously, turning training into big matrix multiplications that GPUs devour. This is the pragmatic reason they scaled.
• Constant path length. Any two tokens interact directly through attention, so long-range dependencies are as easy as short ones.
• It just keeps scaling. Stack more blocks, widen the vectors, feed more data, and performance keeps improving, the empirical observation that launched the era of large language models. BERT (Devlin et al., 2019) kept only the encoder (for understanding) ; the GPT family kept only the decoder (for generation) ; both are the same block from Figure 2, repeated.

Why should stacking these blocks be able to represent the staggering variety of functions language requires ? The reassuring theoretical backdrop is the Universal Approximation Theorem : the feed-forward sub-layers alone are universal approximators, and attention adds the ability to route information between positions. Representational power was never the worry ; as that post stresses, the real magic is that these models can be trained at all.

7. Conclusion

Strip away the vocabulary and the Transformer is short to describe : embed the tokens, add a positional fingerprint, then repeat “let every word look at the others, then let every word think” a handful of times, with residual skips and normalisation to keep the deep stack healthy. Two towers, an encoder that reads and a decoder that writes, cooperate through cross-attention, and a masked softmax lets the whole thing generate text one word at a time. No recurrence, no convolution, attention really was, more or less, all we needed.

References

1. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations (ICLR).
   @inproceedings{Bahdanau2014,
     author = {Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
     title = {Neural Machine Translation by Jointly Learning to Align and Translate},
     booktitle = {International Conference on Learning Representations (ICLR)},
     year = {2015}
   }
   

2. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT, 4171–4186.
   @inproceedings{Devlin2019,
     author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
     title = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
     booktitle = {Proceedings of NAACL-HLT},
     pages = {4171--4186},
     year = {2019}
   }
   

3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS), 30, 5998–6008.
   @inproceedings{Vaswani2017,
     author = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser, {\L}ukasz and Polosukhin, Illia},
     title = {Attention Is All You Need},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     volume = {30},
     pages = {5998--6008},
     year = {2017}
   }
   

1. The paper’s base model uses $N = 6$ blocks, model dimension $d = 512$, $h = 8$ attention heads, and a feed-forward inner size of $2048$. Modern large language models keep the same block but scale these numbers by orders of magnitude. ↩