Back to Media

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Dao, Fu, Ermon, Rudra, Ré
paper completed ai-ml
Year 2022
External Link https://arxiv.org/pdf/2205.14135
attention efficiency GPU optimization from:language-models

Notes

IO-aware attention that is both faster and uses less memory. Essential infrastructure.

View Resource All Media More in Ai-Ml

More in Ai-Ml

An Introduction to Statistical Learning (ISLR2)
James, Witten, Hastie, Tibshirani · book
Artificial Intelligence: A Modern Approach
Russell & Norvig · book
Attention Is All You Need
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin · paper
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, Chang, Lee, Toutanova · paper