June 25, 2024
Reverse-Process Synthetic Data Generation for Math Reasoning
Training LLMs on mathematical reasoning by inverting easy-to-solve problems: generate derivatives, reverse them into integration exercises with full step-by-step solutions.
Machine Learning
Browse posts by category
February 19, 2024
Fine-Tuning a Tiny LLM for ElasticSearch DSL
Fine-tuning a small language model to generate ElasticSearch DSL queries from natural language, as a proof of concept for domain-specific LLM specialization.
March 15, 2026
Superintelligence May Not Require a Breakthrough
The most dramatic possibility in AI might arrive through the most mundane mechanism. Not a beam of sacred light. A sufficiently good build system.
January 15, 2026
From A* to GPT: Rational Agents and the Representation Problem
The classical AI curriculum teaches rational agents as utility maximizers. The progression from search to RL to LLMs is really about one thing: finding representations that make decision-making tractable.
December 3, 2025
Infinigram: Variable-Length N-grams via Suffix Arrays
A corpus-based language model using suffix arrays for O(m log n) pattern matching. The corpus is the model.
October 8, 2025
DreamLog: Logic Programming That Dreams to Improve Itself
A logic programming system that alternates between wake and sleep phases, using LLMs for knowledge generation during wake and compression-based learning during sleep.
January 15, 2025
CTW Experimental Results: Theory Meets Practice
Validating Context Tree Weighting through experiments, including a bug that changed everything.
September 30, 2024
All Induction Is the Same Induction
Solomonoff induction, MDL, speed priors, and neural networks are all special cases of one Bayesian framework with four knobs.
September 20, 2024
Neural Language Models: From RNNs to Transformers
The evolution of neural sequence prediction, and how it connects to classical methods
June 15, 2024
Comparing Prediction Methods: CTW vs. N-grams vs. Neural LMs
The bias-data trade-off in sequential prediction: when to use CTW, n-grams, or neural language models.
March 20, 2024
Instrumental Goals and Hidden Codes in RLHF'd Language Models
How RLHF-trained language models may develop instrumental goals, and the information-theoretic limits on detecting them.
February 19, 2024
Approximations of Solomonoff Induction
I experiment with simple predictive / generative models to approximate Solomonoff induction for a relatively simple synthetic data-generating process.
June 17, 2023
Uses and limits of abstractions
Abstractions let us reason about complex systems despite our cognitive limits. But some systems resist compression entirely.
June 17, 2023
Working Memory as an Inductive Bias
How the limited capacity of human working memory acts as regularization, shaping our reasoning and possibly preventing cognitive overfitting.
August 15, 2016
N-gram Language Models
The classical approach to sequence prediction: counting and smoothing
November 10, 2015
Data Generating Processes for Sequential Prediction
Markov processes and tree sources: understanding where sequences come from
September 20, 2014
The Bayesian Prediction Framework
Model averaging over hypotheses, the principled way to handle uncertainty in prediction
March 15, 2012
Solomonoff Induction: The Theoretical Ideal
The optimal predictor is incomputable. What we can learn from it anyway.
June 1, 2010
Introduction to Sequential Prediction
The problem of predicting what comes next, from compression to language models