June 9, 2026
Bengio's Language Model: the Markov Assumption Made Architectural
The simplest neural language model: embed the last N tokens, concatenate, predict the next. What you give up with a fixed window, and what you gain by dropping recurrence.