Language Models

Speech and Language Processing (3rd ed. draft)

Notes

The canonical NLP book, updated for the LLM era.

The Unreasonable Effectiveness of Recurrent Neural Networks

Notes

Seminal blog post demonstrating char-level RNN power. Shakespeare, LaTeX, kernel code generation.

The Unreasonable Effectiveness of Recurrent Neural Networks

Review

Seminal blog post demonstrating the power of character-level RNNs. Shows Shakespeare generation, Wikipedia generation, LaTeX generation, and Linux kernel code generation. The visualizations of LSTM cells are particularly illuminating.

Infinigram: Corpus-Based Language Modeling via Suffix Arrays with LLM Probability Mixing

Infinigram: Variable-Length N-grams via Suffix Arrays

A corpus-based language model using suffix arrays for O(m log n) pattern matching. The corpus is the model.

Language Calculus: An Algebraic Framework for LLM Composition

A mathematical framework that treats language models as algebraic objects with compositional structure.

Neural Language Models: From RNNs to Transformers

The evolution of neural sequence prediction, and how it connects to classical methods

Comparing Prediction Methods: CTW vs. N-grams vs. Neural LMs

The bias-data trade-off in sequential prediction: when to use CTW, n-grams, or neural language models.

N-gram Language Models

The classical approach to sequence prediction: counting and smoothing