Attention is all you need
The Transformer paper. Drops recurrence entirely, and in doing so makes sequence modelling a parallel problem rather than a sequential one.
The architectural claim is in the title, but the consequence that actually mattered is about hardware, not attention.
What it replaces
Before this, sequence models were recurrent: to compute step t you needed step t−1. That data dependency is a hard serial constraint, and it caps how much of a GPU you can use regardless of how large the model is. The paper removes recurrence and convolution both, and builds the encoder and decoder out of stacked self-attention and position-wise feed-forward layers.
The mechanism
Self-attention lets every position attend to every other position in one operation, so path length between any two tokens is constant rather than linear in their distance. Multi-head attention runs several of these in parallel subspaces, which lets different heads specialise. Because the operation is order-invariant, position has to be injected explicitly — hence positional encodings.
Why it reads as a systems paper
The thing worth keeping: the win came from making the computation parallelisable, which turned model quality into a function of compute you could actually buy. That is a different kind of argument than “this inductive bias is better,” and it is the one that generalised.
Open threads
- Attention is quadratic in sequence length. Everything since has been some negotiation with that fact.
- The geometry question stays open for me — see Loss landscapes and terrain.