---
title: 'Attention is all you need'
date: 2026-10-14
summary: 'The Transformer paper. Drops recurrence entirely, and in doing so makes sequence modelling a parallel problem rather than a sequential one.'
status: growing
topics: [neural-networks]
canonical: 'https://yhzhan.me/notes/attention-is-all-you-need/'
source:
  type: paper
  title: 'Attention Is All You Need'
  author: 'Vaswani et al.'
  url: 'https://arxiv.org/abs/1706.03762'
  length: '15 pp'
  year: 2017
linked_from: [loss-landscapes-and-terrain]
links_to: [loss-landscapes-and-terrain]
---
The architectural claim is in the title, but the consequence that actually
mattered is about hardware, not attention.

## What it replaces

Before this, sequence models were recurrent: to compute step *t* you needed
step *t−1*. That data dependency is a hard serial constraint, and it caps how
much of a GPU you can use regardless of how large the model is. The paper
removes recurrence and convolution both, and builds the encoder and decoder out
of stacked self-attention and position-wise feed-forward layers.

## The mechanism

Self-attention lets every position attend to every other position in one
operation, so path length between any two tokens is constant rather than linear
in their distance. Multi-head attention runs several of these in parallel
subspaces, which lets different heads specialise. Because the operation is
order-invariant, position has to be injected explicitly — hence positional
encodings.

## Why it reads as a systems paper

The thing worth keeping: the win came from making the computation
*parallelisable*, which turned model quality into a function of compute you
could actually buy. That is a different kind of argument than "this inductive
bias is better," and it is the one that generalised.

## Open threads

- Attention is quadratic in sequence length. Everything since has been some
  negotiation with that fact.
- The geometry question stays open for me — see [[loss-landscapes-and-terrain]].