Back to Media

Scaling Laws for Neural Language Models

Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei
paper completed ai-ml
Year 2020
External Link https://arxiv.org/pdf/2001.08361
scaling laws compute loss prediction from:language-models

Notes

Power-law relationships between compute, data, parameters, and loss. Empirical scaling science.

View Resource All Media More in Ai-Ml

More in Ai-Ml

An Introduction to Statistical Learning (ISLR2)
James, Witten, Hastie, Tibshirani · book
Artificial Intelligence: A Modern Approach
Russell & Norvig · book
Attention Is All You Need
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin · paper
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, Chang, Lee, Toutanova · paper