Inductive Biases in Neural Networks
A from-scratch monograph on why each neural architecture works, built on a pure-Python library you can read. One lens, inductive bias along three axes (the architecture, the output head, and whether training can actually find the solution), applied end to end: from why a single linear unit cannot learn XOR through to reverse-engineering a small transformer with causal tracing, and on to reinforcement learning. Pedagogical, not new research. The code for every figure and claim is in the library.
This is the bound, systematic version of the Inductive Biases in Neural Networks series. Eight chapters, three parts, built on the pure-Python scratchnn library. It is pedagogical, not new research: there is no theorem here that was not already known. What it offers is one lens applied all the way through, with the hand-derived backward passes, the actual code, and the experiments the posts only gesture at.
A chapter-by-chapter HTML edition, rendered from the LaTeX source, is in progress and will be embedded here. For now the PDF is below.
Discussion & Related
Attention Weight Is Not Information Flow
The trained pointer model reads exactly the right memory cell, provably. Its attention barely shows where. The gap, and the causal probe that closes it.
June 9, 2026 · 4 min read