Neural Networks
Architectures, optimisation, and what a loss landscape has in common with a mountain.
2 notes · 0 evergreen · 1 growing · 1 seed
The part I keep coming back to is not the architectures but the geometry: high-dimensional surfaces, and the question of why gradient descent finds anything useful on them at all.
NotesIn the graph →
- Attention is all you needThe Transformer paper. Drops recurrence entirely, and in doing so makes sequence modelling a parallel problem rather than a sequential one.
- Loss landscapes and terrainSeed. The slope-reading intuition from backcountry skiing keeps wanting to be a statement about optimisation geometry, and I have not made it rigorous yet.