Attention

Attention Is a Learned Pointer Dereference

An attention head is a learned content-addressable lookup: a query matches keys, retrieves a value, exactly like dereferencing a pointer. Depth is how many lookups you can compose.

Attention Is All You Need

Notes

Introduced the Transformer architecture. The paper that started everything.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Notes

IO-aware attention that is both faster and uses less memory. Essential infrastructure.

Neural Language Models: From RNNs to Transformers

The evolution of neural sequence prediction, and how it connects to classical methods