The idea

Some problems are easy in one direction and hard in the other. Taking a derivative is mechanical. Finding an antiderivative can require genuine creativity. Generating a random expression and verifying a proof is easy. Discovering the proof is hard.

RPSDG (Reverse-Process Synthetic Data Generation) exploits this asymmetry. Run the easy direction with full step-by-step work, then reverse the result to get a hard problem with a known solution. You end up with process-supervised training data: not just the answer, but the entire derivation.

Richard Sutton’s “The Bitter Lesson” argues that methods scaling with compute and data will eventually win. The bottleneck is high-quality data. A lot of the world’s data is latent, the processes that generated it are not written down. In math, the way a proof was discovered is usually hidden behind a polished presentation. RPSDG is one way to manufacture that hidden process data.

Derivatives to Integrals

Computing derivatives is mechanical. Integration often is not. That asymmetry gives us a data pipeline.

1. Start with known functions. Pick functions \( f(x) \) with closed-form derivatives: polynomials, trig, exponentials, logarithms. Vary complexity.

2. Differentiate with full work shown. Take the derivative of \( f(x) \) to get \( f'(x) \), recording every step.

3. Reverse the process. Now \( f'(x) \) is the problem and \( f(x) \) is the solution. The recorded steps, read backward, give you a worked integration example.

By composing functions of varying complexity, you get integration problems of graduating difficulty. The training data comes with step-by-step solutions for free, because you generated it by running the easy direction.

Proofs by Random Walk

The same idea works for theorem proving. Generating proofs is hard. Verifying them is (comparatively) easy.

1. Random walks in expression space. Start with a random expression \( e_{\text{start}} \). Apply rewrite rules \( r_1, r_2, \ldots, r_n \) to get a chain of intermediate expressions ending at \( e_{\text{end}} \).

2. Read off the theorem. The pair \( (e_{\text{start}}, e_{\text{end}}) \) is a theorem. The chain of rewrites is its proof.

3. Reverse when useful. Running the chain backward works too, especially when a complex step in one direction (integration) becomes simple in the other (differentiation).

4. Scale it. Random starting points and random rewrite sequences give you a diverse set of theorems and proofs automatically. No human has to come up with the theorem first.

What this gets you

The training data has process supervision baked in. Every example includes intermediate steps, not just the final answer. That should help LLMs learn multi-step reasoning rather than pattern-matching to answers.

It also gives you something like explainability for free: the model’s training data literally consists of step-by-step solutions, so the model has a better chance of producing step-by-step reasoning at inference time.

Limitations and next steps

This is early work. The data generation pipeline exists, but I have not yet run the full fine-tuning experiments and benchmarks. The planned pipeline is:

• Data generation with graduating difficulty (curriculum learning)
• Fine-tuning transformer-based LMs on the generated data
• Self-supervised learning experiments
• Evaluation against standard math reasoning benchmarks

Further out, I want to explore reinforcement learning for rewarding multi-step reasoning even when the solution is not known in advance but can be verified.

Related

The verification asymmetry exploited here shows up at multiple levels. At inference time, it enables tree search over LLM solutions: I Spent $0.48 to Find Out When MCTS Actually Works for LLM Reasoning. At the level of scientific method, it’s the core of Science as Verifiable Search: cheap testing enables fast iteration through hypothesis space.

I demoed two projects:

1. A simple Colab notebook using basic Python to generate text with an n-gram model. The point was to illustrate the core idea behind language models and show concretely why n-grams fall short, which motivates the transformer architecture.

2. A project that uses ElasticSearch and LLMs to allow natural language search queries over databases.

The talk went well. Content and code are on GitHub.

The Problem

In reliability engineering, you often find yourself in an annoying situation: a system fails, but you do not know which component caused the failure. This is called masked failure data. On top of that, some systems are still running when you stop observing them, so you only know they survived at least that long. That is right censoring. Both are common in practice. Identifying the exact failed component is expensive or sometimes impossible.

The project builds a likelihood-based framework that handles both masking and censoring simultaneously, models component lifetimes with Weibull distributions, derives closed-form Fisher information for the exponential special case, and provides bootstrap methods for uncertainty quantification. I implemented it all in an R package so practitioners can actually use it.

Related Work

This connects to several other posts and projects:

• Closed-Form Results for Masked Exponential Series Systems covers the exponential distribution special case with analytical solutions
• likelihood.model R package is the software implementation

See the full project page here.

This seemingly simple idea has profound implications. In this post, I’ll show how to build a content-addressed virtual filesystem in Python, exploring how immutability and content addressing work together to create elegant, functional data structures.

Why Content Addressing?

Consider a traditional filesystem. When you modify a file, the system overwrites the old content. The file’s identity (its path) stays the same, but its content changes. This mutable approach has problems:

1. No automatic history: Once you overwrite, the old data is gone
2. No deduplication: Two identical files take up twice the space
3. No integrity verification: Corruption can go undetected

Content addressing solves all three. If a file’s identity is its content (via a hash), then:

1. History is preserved: Changing content creates a new hash, so the old version still exists
2. Deduplication is automatic: Identical content has identical hashes—stored once
3. Integrity is built-in: If the content doesn’t match the hash, you know something’s wrong

The Node Hierarchy

Let’s build this. First, we define our filesystem nodes using Python’s frozen dataclasses:

from dataclasses import dataclass, field
import hashlib
import json

@dataclass(frozen=True)
class Node:
    """Base class for all filesystem nodes."""
    mode: int
    uid: int = 1000
    gid: int = 1000
    mtime: float = field(default_factory=time.time)

    def compute_hash(self) -> str:
        """Compute SHA256 hash of this node including all metadata."""
        data = json.dumps(self.to_dict(), sort_keys=True)
        return hashlib.sha256(data.encode()).hexdigest()

The frozen=True parameter is crucial. It makes instances immutable—you cannot modify a Node after creation. Any “change” requires creating a new Node.

We then specialize for different node types:

@dataclass(frozen=True)
class FileNode(Node):
    """Regular file node."""
    content: bytes = b""

@dataclass(frozen=True)
class DirNode(Node):
    """Directory node containing references to child nodes."""
    children: Dict[str, str] = field(default_factory=dict)  # name -> hash

Notice that DirNode.children maps names to hashes, not to Node objects directly. This is the key insight: directories don’t contain files; they contain references to file hashes. The actual nodes live in a separate store.

The DAG Structure

This reference-based approach creates a Directed Acyclic Graph (DAG):

class FileSystem:
    """Content-addressable virtual filesystem."""

    def __init__(self):
        # The DAG: hash -> Node
        self.nodes: Dict[str, Node] = {}

        # Path index: absolute path -> hash
        self.paths: Dict[str, str] = {}

    def _add_node(self, node: Node) -> str:
        """Add a node to the DAG, returning its hash."""
        node_hash = node.compute_hash()
        if node_hash not in self.nodes:
            self.nodes[node_hash] = node
        return node_hash

When we add a node, we compute its hash and store the mapping hash → node. If an identical node already exists (same hash), we don’t duplicate it—we just return the existing hash. Deduplication is automatic.

Immutable Updates

Here’s where immutability shines. When we write to a file, we don’t modify anything. Instead, we:

1. Create a new FileNode with the new content
2. Create a new DirNode for the parent, pointing to the new file hash
3. Update the path index

def write(self, path: str, content: bytes) -> bool:
    """Write content to a file."""
    parent_path, name = self._get_parent_path(path)
    parent_hash = self.paths[parent_path]
    parent = self.nodes[parent_hash]

    # Create new file node
    file_node = FileNode(content)
    file_hash = self._add_node(file_node)

    # Create new parent directory with updated child reference
    new_children = dict(parent.children)
    new_children[name] = file_hash
    new_parent = DirNode(children=new_children)
    new_parent_hash = self._add_node(new_parent)

    # Update path index
    self.paths[parent_path] = new_parent_hash
    self.paths[path] = file_hash

    return True

The old FileNode still exists in self.nodes. The old DirNode still exists too. We’ve just created new versions and updated where the path points. This is structural sharing—unchanged parts of the tree are shared between versions.

Visualizing the DAG

Let’s trace through an example:

fs = FileSystem()
fs.mkdir("/project")
fs.write("/project/main.py", b"print('hello')")
fs.write("/project/main.py", b"print('world')")

After these operations, our DAG contains:

Hash: a1b2c3... → DirNode(children={})           # original /project
Hash: d4e5f6... → FileNode("print('hello')")     # first version
Hash: g7h8i9... → DirNode(children={"main.py": "d4e5f6..."})
Hash: j0k1l2... → FileNode("print('world')")     # second version
Hash: m3n4o5... → DirNode(children={"main.py": "j0k1l2..."})

Both versions of main.py exist. The path /project/main.py points to the latest hash (j0k1l2...), but we could easily restore the old version if we tracked which hashes corresponded to which versions.

Benefits in Practice

This design enables powerful features almost for free:

Snapshots: Save the current paths dictionary. Restore it later to go back in time.

def snapshot(self) -> Dict[str, str]:
    """Create a snapshot of the current filesystem state."""
    return dict(self.paths)

def restore(self, snapshot: Dict[str, str]):
    """Restore filesystem to a previous snapshot."""
    self.paths = dict(snapshot)

Deduplication: Multiple paths can point to the same hash.

# These might share the same underlying node if content is identical
fs.write("/file1.txt", b"hello")
fs.write("/file2.txt", b"hello")  # Same hash, no new storage

Integrity checking: If someone asks for a file, we can verify it.

def verify(self, path: str) -> bool:
    """Verify a file's integrity."""
    node_hash = self.paths[path]
    node = self.nodes[node_hash]
    return node.compute_hash() == node_hash

The Functional Programming Connection

This approach is deeply connected to functional programming. In FP:

• Data is immutable
• “Changes” create new values
• Sharing is safe because nothing mutates

Our filesystem follows these principles exactly. Nodes are frozen. “Writing” creates new nodes. Multiple paths can safely share nodes because nodes never change.

This is why Clojure’s persistent data structures, Haskell’s pure values, and Git’s object store all use similar ideas. Content addressing + immutability = safe, efficient, verifiable data.

Trade-offs

Nothing is free. This approach has costs:

1. Memory: Old versions accumulate. You need garbage collection to reclaim space from unreachable nodes.

2. Performance: Creating new nodes for every change can be slower than in-place mutation for write-heavy workloads.

3. Complexity: Path resolution requires extra indirection through the hash table.

For many use cases—especially those valuing history, integrity, and safe concurrency—these trade-offs are worthwhile.

Conclusion

Content addressing transforms how we think about data. Instead of “where is this file?” we ask “what is this content’s identity?” Instead of destructive updates, we create new versions while sharing unchanged structure.

This pattern appears everywhere: Git, IPFS, Nix, Docker layers, and many database internals. Understanding it opens doors to building robust, elegant systems.

The full implementation in DagShell extends these ideas with a complete POSIX-like interface, demonstrating how content addressing can underpin a full virtual filesystem.

Next in this series: Unix Philosophy in Python — building composable commands with method chaining.

The idea: take a task that large models handle well (translating “find all orders over $100 from last month” into the corresponding ElasticSearch JSON query) and see if a tiny model can learn it from synthetic data.

The data pipeline works like this. I started by generating examples from GPT-4. Then I wrote a script that samples from those outputs and uses them as few-shot examples for Mistral, which generates a much larger synthetic dataset. The next step is reshaping the data into the expected format and fine-tuning.

This is early stage. The synthetic data exists, the fine-tuning has not happened yet. I will update this post with results.

If you are interested in collaborating on this or related projects, email me at lex@metafunctor.com.

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

This philosophy gave us pipes (|), small focused utilities (cat, grep, sort), and the ability to compose complex operations from simple parts:

cat access.log | grep "404" | cut -d' ' -f1 | sort | uniq -c | sort -rn

Can we bring this composability to Python? Not just subprocess calls, but native Python objects that pipe and chain as naturally as Unix commands? Let’s build it.

The Problem with Methods

Standard Python methods have a composition problem. Consider:

# We want to: read a file, filter lines, count words
content = read_file("/data/log.txt")
filtered = grep(content, "ERROR")
count = wc(filtered)

This works, but it’s inside-out compared to how we think about it. We think “read, then filter, then count”—a left-to-right pipeline. But we write it bottom-up with intermediate variables.

Method chaining helps:

read_file("/data/log.txt").grep("ERROR").wc()

Now it flows left-to-right. But how do we make arbitrary operations chainable?

The CommandResult Pattern

The key insight is wrapping every result in a chainable container:

@dataclass
class CommandResult:
    """Wrapper that enables method chaining."""
    data: Any           # The actual result data
    text: str = None    # Text representation
    exit_code: int = 0  # Unix-style exit code
    _shell: 'DagShell' = None  # Reference back to the shell

    def __str__(self) -> str:
        if self.text is not None:
            return self.text
        return str(self.data)

Every command returns a CommandResult. The result carries the data and a reference to the shell that produced it. This reference enables chaining—we can call more methods through it.

Making Commands Chainable

Here’s a simple command implementation:

class DagShell:
    def echo(self, *args) -> CommandResult:
        """Echo arguments to output."""
        text = ' '.join(str(arg) for arg in args)
        return CommandResult(
            data=text,
            text=text,
            exit_code=0,
            _shell=self
        )

The magic happens in CommandResult. We add methods that delegate back to the shell:

@dataclass
class CommandResult:
    # ... fields as before ...

    def grep(self, pattern: str) -> 'CommandResult':
        """Filter lines matching pattern."""
        return self._shell.grep(pattern, input_data=self.data)

    def wc(self, *flags) -> 'CommandResult':
        """Count lines, words, or characters."""
        return self._shell.wc(*flags, input_data=self.data)

    def out(self, path: str) -> 'CommandResult':
        """Redirect output to a file."""
        self._shell.fs.write(path, str(self).encode())
        return self

Now we can chain:

shell.echo("hello world").wc("-w")  # Returns: 2
shell.cat("/data/log.txt").grep("ERROR").wc("-l")  # Count error lines

The Dual Nature: Objects and Files

Unix commands have a superpower: the same output can go to the screen, a file, or another command. We can replicate this:

# Chain more commands
result = shell.cat("/data/log.txt").grep("ERROR")

# Get as Python object
lines = result.lines()  # List[str]

# Or redirect to a file
result.out("/data/errors.txt")

The CommandResult is simultaneously:

1. A Python object you can inspect and manipulate
2. A text stream you can redirect to files
3. An input source for the next command in a pipeline

This dual nature—structured data and text stream—bridges the gap between Unix philosophy and Python’s object orientation.

Implementing Piping

True Unix pipes pass data between processes. We simulate this with a “last result” mechanism:

class DagShell:
    def __init__(self):
        self._last_result: Optional[CommandResult] = None

    def _(self) -> CommandResult:
        """Return the last command's result (like $? or $_)."""
        if self._last_result is None:
            return CommandResult(data='', text='', exit_code=0, _shell=self)
        return self._last_result

Now we can build pipelines:

shell.cat("/data/log.txt")
shell._().grep("ERROR")
shell._().wc("-l")

Or more elegantly, commands can accept piped input:

def grep(self, pattern: str, input_data=None) -> CommandResult:
    """Filter lines matching pattern."""
    if input_data is None:
        input_data = self._last_result.data if self._last_result else ''

    lines = str(input_data).splitlines()
    matching = [line for line in lines if pattern in line]

    result = CommandResult(
        data=matching,
        text='\n'.join(matching),
        exit_code=0 if matching else 1,
        _shell=self
    )
    self._last_result = result
    return result

Method Chaining in Action

Let’s build something real—a log analysis pipeline:

# Create a shell and some test data
shell = DagShell()
shell.mkdir("/logs")
shell.echo("""
2024-01-15 10:30:00 INFO  User login: alice
2024-01-15 10:31:00 ERROR Database connection failed
2024-01-15 10:32:00 INFO  User login: bob
2024-01-15 10:33:00 ERROR Timeout waiting for response
2024-01-15 10:34:00 WARN  High memory usage
2024-01-15 10:35:00 ERROR Disk space low
""".strip()).out("/logs/app.log")

# Pipeline: find errors, extract timestamps, save to file
(shell
    .cat("/logs/app.log")
    .grep("ERROR")
    .cut(delimiter=" ", fields="1,2")
    .out("/logs/error_times.txt"))

# Read the result
print(shell.cat("/logs/error_times.txt"))
# Output:
# 2024-01-15 10:31:00
# 2024-01-15 10:33:00
# 2024-01-15 10:35:00

Each step does one thing. The chain composes them into a useful operation. The data flows left-to-right, just like our mental model.

Directory Navigation with a Stack

Unix has cd, but shells also have pushd and popd for directory stacks. We implement this:

class DagShell:
    def __init__(self):
        self._cwd = '/'
        self._dir_stack = []

    def pushd(self, path: str) -> CommandResult:
        """Push current directory and change to new one."""
        self._dir_stack.append(self._cwd)
        return self.cd(path)

    def popd(self) -> CommandResult:
        """Pop directory from stack and change to it."""
        if not self._dir_stack:
            return CommandResult(data='', text='popd: directory stack empty', exit_code=1)
        old_dir = self._dir_stack.pop()
        return self.cd(old_dir)

Now we can navigate without losing our place:

shell.pushd("/project/src")
# ... work in src ...
shell.pushd("tests")
# ... work in tests ...
shell.popd()  # back to /project/src
shell.popd()  # back to original directory

Exit Codes: Success and Failure

Unix commands return exit codes: 0 for success, non-zero for failure. We include this in CommandResult:

def grep(self, pattern: str) -> CommandResult:
    # ... filtering logic ...
    return CommandResult(
        data=matching,
        text='\n'.join(matching),
        exit_code=0 if matching else 1,  # 1 if no matches
        _shell=self
    )

This enables conditional logic:

result = shell.grep("pattern", file="/data/log.txt")
if result.exit_code == 0:
    print(f"Found {len(result.lines())} matches")
else:
    print("No matches found")

The Philosophy Applied

Let’s revisit McIlroy’s principles and see how we’ve applied them:

“Do one thing well”: Each method (cat, grep, wc, cut) does exactly one thing.

“Work together”: CommandResult enables any command to connect to any other.

“Text streams as universal interface”: Every result has a text representation via __str__, making it redirectable and pipeable.

We’ve also added Python-specific benefits:

• Type safety: Results carry structured data, not just text
• Introspection: result.lines(), result.data, etc.
• Chaining: Method chains read left-to-right like pipelines

The Fluent Pattern

This is an instance of the Fluent Interface pattern, where methods return self (or a related object) to enable chaining. Martin Fowler described it in 2005, but the idea is older—Smalltalk embraced it from the beginning.

The key is designing methods that:

1. Perform their action
2. Return something chainable
3. Maintain enough context for the next operation

When done well, code reads almost like prose:

(shell
    .mkdir("/project")
    .cd("/project")
    .echo("# My Project")
    .out("README.md")
    .echo("def main(): pass")
    .out("main.py")
    .ls("-la"))

Trade-offs

This approach isn’t free:

1. Wrapper overhead: Every result is wrapped in CommandResult
2. Learning curve: Users must understand the chaining pattern
3. Debugging: Long chains can be hard to debug—where did it fail?

For scripting and exploratory work, the expressiveness outweighs these costs. For performance-critical code, you might unwrap to raw operations.

Conclusion

The Unix philosophy isn’t about Unix—it’s about composability. Small, focused operations that connect through a universal interface create systems greater than the sum of their parts.

In Python, we achieve this with:

• Wrapper types that carry data and context
• Method chaining for left-to-right flow
• Dual representations as objects and text streams
• Exit codes for success/failure signaling

The result is a fluent interface where complex operations emerge from simple, composable parts—Unix philosophy, realized in Python.

See the full implementation in DagShell, which builds a complete virtual filesystem with this fluent pattern.

Next in this series: Embedding a Scheme Interpreter — adding a DSL for filesystem scripting.

But embedding a full language runtime is heavy. What if we just need something small—variables, functions, conditionals, loops? A Lisp-family language is perfect for this: the syntax is trivial to parse, the semantics are clean, and it’s powerful enough for real scripting.

In this post, I’ll walk through building a minimal Scheme interpreter in Python and integrating it with a virtual filesystem. We’ll cover the complete pipeline: tokenization, parsing, evaluation, and extension with custom primitives.

Why Scheme?

Scheme is a minimalist Lisp dialect. Its syntax is just parentheses and atoms:

(define x 10)
(+ x (* 2 3))
(if (> x 5) "big" "small")

This uniformity makes parsing trivial. There’s no operator precedence, no complex grammar—just:

1. Atoms: numbers, strings, symbols
2. Lists: (thing thing thing ...)

That’s it. The entire parser can be written in under 50 lines.

Step 1: Tokenization

First, we convert source code into tokens. Scheme’s tokenization is simple: split on whitespace, but respect strings and parentheses.

def tokenize(text: str) -> List[str]:
    """Convert Scheme code into tokens."""
    # Handle comments (lines starting with ;)
    lines = text.split('\n')
    text = '\n'.join(line.split(';')[0] for line in lines)

    # Add spaces around parens for easy splitting
    text = text.replace('(', ' ( ').replace(')', ' ) ')

    # Split, handling string literals
    tokens = []
    in_string = False
    current = []

    for char in text:
        if char == '"':
            in_string = not in_string
            current.append(char)
        elif in_string:
            current.append(char)
        elif char.isspace():
            if current:
                tokens.append(''.join(current))
                current = []
        else:
            current.append(char)

    if current:
        tokens.append(''.join(current))

    return tokens

Input: (define x (+ 1 2)) Output: ['(', 'define', 'x', '(', '+', '1', '2', ')', ')']

Step 2: Parsing

Parsing converts tokens into an Abstract Syntax Tree (AST). In Scheme, the AST is just nested Python lists:

@dataclass
class Symbol:
    """Represents a Scheme symbol."""
    name: str

def parse(tokens: List[str]) -> Any:
    """Parse tokens into an AST."""
    def parse_expr(index: int) -> Tuple[Any, int]:
        token = tokens[index]

        if token == '(':
            # Parse a list
            lst = []
            index += 1
            while tokens[index] != ')':
                expr, index = parse_expr(index)
                lst.append(expr)
            return lst, index + 1

        elif token == ')':
            raise SyntaxError("Unexpected )")

        else:
            # Parse an atom
            return parse_atom(token), index + 1

    expr, _ = parse_expr(0)
    return expr

def parse_atom(token: str) -> Any:
    """Parse a single atom."""
    # Try integer
    try:
        return int(token)
    except ValueError:
        pass

    # Try float
    try:
        return float(token)
    except ValueError:
        pass

    # String literal
    if token.startswith('"') and token.endswith('"'):
        return token[1:-1]

    # Boolean
    if token == '#t':
        return True
    if token == '#f':
        return False

    # Symbol
    return Symbol(token)

Input tokens: ['(', 'define', 'x', '(', '+', '1', '2', ')', ')'] Output AST: [Symbol('define'), Symbol('x'), [Symbol('+'), 1, 2]]

The beauty of Lisp: the AST is the syntax. There’s no separate tree structure—it’s just lists of symbols and values.

Step 3: The Environment

Before evaluation, we need an environment to track variable bindings. This is where lexical scoping lives:

class Environment:
    """Lexical environment for variable bindings."""

    def __init__(self, parent: Optional['Environment'] = None):
        self.bindings: Dict[str, Any] = {}
        self.parent = parent

    def define(self, name: str, value: Any):
        """Define a new binding in this environment."""
        self.bindings[name] = value

    def get(self, name: str) -> Any:
        """Look up a binding, checking parent scopes."""
        if name in self.bindings:
            return self.bindings[name]
        elif self.parent:
            return self.parent.get(name)
        else:
            raise NameError(f"Undefined variable: {name}")

    def set(self, name: str, value: Any):
        """Update an existing binding."""
        if name in self.bindings:
            self.bindings[name] = value
        elif self.parent:
            self.parent.set(name, value)
        else:
            raise NameError(f"Undefined variable: {name}")

Each function call creates a new Environment with the current one as its parent. This chain enables closures—inner functions that remember their enclosing scope.

Step 4: Evaluation

The evaluator is the heart of the interpreter. It walks the AST and computes values:

def evaluate(expr: Any, env: Environment) -> Any:
    """Evaluate an expression in an environment."""

    # Self-evaluating: numbers, strings, booleans
    if isinstance(expr, (int, float, str, bool, type(None))):
        return expr

    # Variable lookup
    if isinstance(expr, Symbol):
        return env.get(expr.name)

    # Must be a list (function call or special form)
    if not isinstance(expr, list) or not expr:
        return expr

    op = expr[0]

    # Special forms
    if isinstance(op, Symbol):
        if op.name == 'quote':
            return expr[1]

        if op.name == 'define':
            name = expr[1]
            value = evaluate(expr[2], env)
            env.define(name.name, value)
            return value

        if op.name == 'if':
            condition = evaluate(expr[1], env)
            if condition:
                return evaluate(expr[2], env)
            elif len(expr) > 3:
                return evaluate(expr[3], env)
            return None

        if op.name == 'lambda':
            params = expr[1]
            body = expr[2]
            return Procedure(params, body, env)

    # Function application
    func = evaluate(op, env)
    args = [evaluate(arg, env) for arg in expr[1:]]
    return func(*args)

Special forms (define, if, lambda, etc.) have custom evaluation rules. Everything else is a function call: evaluate the operator, evaluate the arguments, call the function.

Step 5: User-Defined Functions

The lambda form creates procedures. A Procedure captures its parameters, body, and defining environment:

@dataclass
class Procedure:
    """A user-defined function."""
    params: List[Symbol]
    body: Any
    env: Environment

    def __call__(self, *args):
        # Create a new environment for the call
        local_env = Environment(parent=self.env)

        # Bind parameters to arguments
        for param, arg in zip(self.params, args):
            local_env.define(param.name, arg)

        # Evaluate the body in this new environment
        return evaluate(self.body, local_env)

When called, a Procedure:

1. Creates a new environment with its defining environment as parent (closure!)
2. Binds parameters to the passed arguments
3. Evaluates its body in this new environment

This simple structure gives us closures, higher-order functions, and lexical scoping.

Step 6: Built-in Primitives

The global environment provides built-in functions:

def create_global_env() -> Environment:
    """Create environment with built-in primitives."""
    env = Environment()

    # Arithmetic
    env.define('+', lambda *args: sum(args))
    env.define('-', lambda a, b=None: -a if b is None else a - b)
    env.define('*', lambda *args: reduce(lambda x, y: x * y, args, 1))
    env.define('/', lambda a, b: a / b)

    # Comparison
    env.define('=', lambda a, b: a == b)
    env.define('<', lambda a, b: a < b)
    env.define('>', lambda a, b: a > b)

    # List operations
    env.define('car', lambda lst: lst[0])
    env.define('cdr', lambda lst: lst[1:])
    env.define('cons', lambda a, b: [a] + list(b))
    env.define('list', lambda *args: list(args))
    env.define('null?', lambda lst: lst == [])

    # Higher-order functions
    env.define('map', lambda f, lst: [f(x) for x in lst])
    env.define('filter', lambda f, lst: [x for x in lst if f(x)])
    env.define('reduce', lambda f, lst, init: reduce(f, lst, init))

    return env

Each primitive is just a Python function. The interpreter treats them identically to user-defined procedures.

Step 7: Filesystem Integration

Now the interesting part: extending Scheme with filesystem primitives.

def create_global_env(shell=None) -> Environment:
    env = Environment()

    # ... standard primitives ...

    # Filesystem primitives (if shell provided)
    if shell:
        env.define('ls', lambda path='/': shell.ls(path).lines())
        env.define('cat', lambda path: shell.cat(path).text)
        env.define('mkdir', lambda path: shell.mkdir(path) and path)
        env.define('write', lambda path, content:
            shell.fs.write(path, content.encode()) and path)
        env.define('exists?', lambda path: shell.fs.exists(path))
        env.define('cd', lambda path: shell.cd(path) and path)
        env.define('pwd', lambda: shell._cwd)

    return env

Now Scheme can script filesystem operations:

; Create a project structure
(mkdir "/project")
(mkdir "/project/src")
(mkdir "/project/tests")

(write "/project/README.md" "# My Project\n")
(write "/project/src/main.py" "def main(): pass\n")

; List files
(ls "/project")
; => ("README.md" "src" "tests")

; Check existence
(if (exists? "/project/README.md")
    "Found it!"
    "Not found")

Step 8: The REPL

A Read-Eval-Print Loop ties it together:

class SchemeREPL:
    def __init__(self, shell=None):
        self.env = create_global_env(shell=shell)

    def eval_string(self, code: str) -> Any:
        """Evaluate a string of Scheme code."""
        tokens = tokenize(code)
        if not tokens:
            return None

        result = None
        idx = 0

        while idx < len(tokens):
            expr, idx = self._parse_one(tokens, idx)
            result = evaluate(expr, self.env)

        return result

    def run(self):
        """Interactive REPL."""
        print("Scheme REPL (type 'quit' to exit)")
        while True:
            try:
                code = input("scheme> ")
                if code.strip() == 'quit':
                    break
                result = self.eval_string(code)
                if result is not None:
                    print(format_value(result))
            except Exception as e:
                print(f"Error: {e}")

Example Session

Here’s what a session looks like:

scheme> (define double (lambda (x) (* x 2)))
scheme> (double 21)
42

scheme> (map double (list 1 2 3 4 5))
(2 4 6 8 10)

scheme> (mkdir "/data")
"/data"

scheme> (write "/data/numbers.txt" "1\n2\n3\n")
"/data/numbers.txt"

scheme> (cat "/data/numbers.txt")
"1\n2\n3\n"

scheme> (filter (lambda (x) (> x 1))
               (map (lambda (s) (string->number s))
                    (string-split (cat "/data/numbers.txt") "\n")))
(2 3)

We’ve built a scriptable filesystem with a dozen lines of primitives!

Design Patterns

Several patterns make this work:

Homoiconicity: Code is data. The AST is just lists, so we can generate and manipulate code programmatically.

Closures: Functions capture their environment, enabling powerful patterns like partial application and callbacks.

Extension via primitives: Adding features is just adding functions to the global environment. No interpreter changes needed.

Separation of concerns: Tokenizing, parsing, and evaluating are distinct phases with clean interfaces.

Trade-offs

This simple interpreter has limitations:

1. No tail-call optimization: Deep recursion will overflow the Python stack.
2. No macros: We can’t extend the language’s syntax.
3. Error messages are basic: Line numbers would require more tracking.
4. Performance: Pure interpretation is slow for heavy computation.

For a DSL extending an application, these trade-offs are often acceptable. The simplicity enables understanding and modification.

Conclusion

Embedding a language isn’t black magic. At its core:

1. Tokenize: Break text into tokens
2. Parse: Build an AST (for Lisp, just nested lists)
3. Evaluate: Walk the AST, computing values
4. Extend: Add primitives that call into your application

Scheme’s minimal syntax makes this especially tractable. In a few hundred lines, we get variables, functions, closures, conditionals, recursion, and extensibility.

The result is a scriptable application where users can automate, experiment, and build beyond what the core interface provides—exactly what we want from a DSL.

The complete implementation is in DagShell, demonstrating Scheme as a filesystem scripting language alongside Python’s fluent API.

This is the final post in the series. Previous posts: Immutable Content-Addressed Filesystems and Unix Philosophy in Python.

Author search: oeis.org/search?q=alex+towell.

Authored sequences

These sequences were created and first populated by me.

A394445 is backed by a closed-form theorem (a proof that the distinct 2-color Rado number for x+y=kz follows an explicit parity-dependent formula for all k >= 8), with a 500-term b-file. A394661 and A395644 come from a census of the KnotInfo prime-knot tables; A395521 from an exhaustive sandpile-group computation.

Extended sequences

Classical Zarankiewicz-problem sequences (originally by N. J. A. Sloane) where I computed and added new terms past the known frontier.

The Zarankiewicz extensions use SAT: a satisfying assignment exhibits a dense matrix (a lower bound), and an unsatisfiable instance proves the matching upper bound. The most recent terms required a double-lex symmetry-breaking encoding to make the upper-bound proof tractable: the plain encoding could not settle z(10,11;3,4) in seven days, while the symmetry-broken version proved it in 27 minutes.

In review (June 2026)

• A006615 a(11)=79, A006622 a(10)=73, A006625 a(10)=79 (the three exact Zarankiewicz values above), proposed June 5 2026.

The A394445 b-file (500 terms) is already approved and live.

Reviewers

These submissions were reviewed and approved by OEIS editors including Michel Marcus, Sean A. Irvine, Max Alekseyev, and Jon E. Schoenfield.

The OEIS has been a shared ledger of integer sequences since 1964. Adding a term is a small, permanent contribution to a commons that outlasts any one project: a number that was unknown is now known, checked, and citable by anyone who needs it.

Here is a number that bothered me for a week. I wanted the exact value of a small combinatorial quantity called z(10,11;3,4). A SAT solver found a good configuration in about two seconds, which told me the answer was at least 72. To finish, I needed the other direction: a proof that nothing does better, that the answer is at most 72. I started that second SAT job and walked away. It ran for more than seven days without an answer.

Then I added one idea, an old and standard one, and the same proof finished in 27 minutes.

The idea is symmetry breaking. This post builds it up from scratch, watches it work on the problem above (the payoff was three new values in the On-Line Encyclopedia of Integer Sequences), and then watches it fail on a different problem, because knowing when a tool stops working is most of the skill of using it.

A problem you can hold in your hand

The quantity z(m,n;s,t) is the largest number of 1s you can put in an m-by-n grid of 0s and 1s without creating an all-ones s-by-t block. By “block” I mean: pick some s rows and some t columns; the s-by-t cells where they cross must not all be 1.

Picture a 4-by-4 grid. If rows 1 and 3 and columns 2 and 4 all carry 1s at their four intersections, that is a forbidden 2-by-2 block. Avoid every such block, pack in as many 1s as you can, and the maximum is the Zarankiewicz number. These numbers grow in a ragged, formula-resistant way, which is exactly why people compute them one at a time.

This is a natural fit for a SAT solver. Put one Boolean variable on each cell, true when the cell is 1. For every choice of s rows and t columns, add one clause saying “at least one of these s times t cells is 0.” Add a counting constraint, “at least k cells are 1,” and ask the solver:

• If it answers satisfiable, it hands back a grid with k ones and no forbidden block. So z is at least k.
• If it answers unsatisfiable, no such grid exists, so z is less than k.

To pin the exact value you need both: a witness at k, and an impossibility proof at k+1. The witness side is usually easy. The impossibility side, the UNSAT side, is the expensive half.

Why proving impossibility is hard

The asymmetry is intuitive once you say it out loud. To find a witness, the solver needs one lucky arrangement. To prove impossibility, it must rule out every arrangement. That is already a harder job. But there is a second, sneakier cost, and it is the one that matters here.

The solver keeps rediscovering the same grid wearing different clothes.

Take any valid grid. Swap two of its rows. You get a different assignment of the Boolean variables, but it is plainly the same object: same number of 1s, same blocks present or absent. Swap two columns: same story. The solver does not know this. To the solver, the original and the swapped version are two different points in the search space, each to be explored and refuted from scratch.

Counting the copies

Name the thing that is hurting us. The row permutations and the column permutations form a group that acts on every grid while preserving both what we are maximizing (the count of 1s) and what we are forbidding (the blocks). So the grids come in families, called orbits, and every grid in an orbit is the same object relabeled.

How big is an orbit? For a 10-by-11 grid, you can reorder the rows in 10! ways and the columns in 11! ways:

10! times 11! is about 1.4 times 10^14.

That is roughly a hundred trillion relabelings of a single grid. When the solver is trying to prove that no good grid exists, it is wading through a hundred trillion copies of each candidate it has already dismissed. No wonder the job ran for a week.

The fix: keep one copy of each

The cure is to refuse all but one member of each orbit. If we can tell the solver “only consider the canonical version of each grid,” the search shrinks by the size of the orbits.

The standard canonical choice is the lexicographically smallest relabeling, and the standard way to enforce it cheaply is called double-lex:

• require the rows to be in non-decreasing lexicographic order, and
• require the columns to be in non-decreasing lexicographic order.

Read each row left to right as a string of 0s and 1s; demand row 1 is no larger than row 2, which is no larger than row 3, and so on. Do the same down the columns.

The reason this is allowed, and it is worth saying carefully, is a soundness fact: every orbit contains at least one grid that is both row-sorted and column-sorted. So when we throw away every non-doubly-lex grid, we never throw away an entire orbit. If a good grid existed before, a good doubly-lex grid still exists. Satisfiability is untouched, and so is unsatisfiability. We have deleted only redundant copies, never an actual solution. That is the whole game.

Saying “this row is not bigger” in clauses

A SAT solver speaks in clauses, so we have to express “this 0/1 vector is lexicographically less than or equal to that one” as a pile of ORs. The construction is a small classic, and building it is the kind of thing this blog likes to do by hand.

Compare two vectors a and b, position by position from the left. Carry one extra Boolean per position, an “equal so far” flag. The rule:

• while the prefixes are still equal, the current bits must satisfy a-bit implies b-bit (you are not allowed to be the one that is larger);
• update the “equal so far” flag to stay true only if the current bits match.

In clause form, at each position, with eq meaning equal-so-far:

# if equal so far, then a <= b at this position
(not eq) OR (not a) OR b

# eq_next is true exactly when eq held and the bits matched
# (a handful of clauses define eq_next = eq AND (a == b))

It is linear in the length of the vector, and you can check it against a truth table to be sure (I did, for all vectors up to length four, before trusting it on anything larger). Apply it to each adjacent pair of rows and each adjacent pair of columns, and you have double-lex.

One honest caveat. Double-lex is not a complete symmetry break. Some orbits still keep more than one doubly-lex member, and we only constrained adjacent pairs rather than all permutations directly. It is sound and cheap, not perfect. That is fine. Removing most of the symmetry is enough to turn intractable into tractable, and chasing a unique representative would cost more than it returns.

The payoff, in real seconds

First, sanity. A known value: z(9,9;3,4) is 56. The proof that it is not 57 is a small UNSAT instance. With the plain encoding, the solver did not finish it inside a 90-second budget. With double-lex, it finished in about ten seconds. The symmetry break is the difference between “no answer” and “ten seconds” on a problem whose answer we already knew, which is exactly the kind of validation you want before trusting it on a problem whose answer you do not.

Then the one that started this post:

That 27-minute proof, paired with the two-second witness, settled z(10,11;3,4) = 72 exactly. Two siblings fell the same way. All three went to the OEIS as new values of classic Zarankiewicz sequences (A006615, A006622, A006625), and they sit alongside the rest of my OEIS contributions, which is where this kind of computed-then-verified result tends to land.

Now watch it fail

If the story stopped there, you might think symmetry-broken SAT is a frontier machine. It is not, and the failure is more instructive than the success.

Try a different problem: the most edges you can put on n points without ever forming a 4-cycle (a square). This is also a 0/1 matrix question, an adjacency matrix this time, and it has the same flavor of symmetry, so the same tool should apply. I built it and validated it: it reproduces the known values (OEIS A006855) exactly up to n = 15.

Then it dies. The runtime grows by roughly a factor of five for every extra vertex. By n = 15 a single value already takes about a minute, and the known frontier sits at n = 41. Five to the twenty-sixth power times a minute is not a number of minutes anyone will wait. The tool that cracked Zarankiewicz in 27 minutes cannot get near this frontier at all.

Why the split? It comes down to the shape of the symmetry.

• In the Zarankiewicz grid, rows and columns are separate axes. You can sort them independently, and the dimensions are modest (10 by 11). Adjacent lexicographic order removes enough of the redundancy to win.
• In a graph, there is one symmetry, the relabeling of vertices, and it permutes the rows and the columns of the adjacency matrix together. Adjacent-pair lexicographic order barely dents that single large group, and the leftover symmetry still explodes the search.

The people who actually reach the C4-free frontier do not use plain SAT at all. They use orderly generation, building graphs in a canonical form so that no two relabelings are ever produced (the lineage of nauty and its relatives). That is a different and heavier instrument.

The lesson generalizes. Symmetry-broken SAT scales when the symmetry group is a product of modest factors you can sort independently, and it stalls when it is one big group. Before you reach for it, look at the symmetry you are facing and ask which of those two worlds you are in.

Credit where it is due

None of the machinery here is mine, and that is worth stating plainly rather than burying. Double-lex symmetry breaking for matrix models is from Flener, Frisch, Hnich, Kiziltan, Miguel, Pearson, and Walsh, “Breaking row and column symmetries in matrix models” (CP 2002). Attacking Zarankiewicz numbers with symmetry-aware SAT was done by Jeremy Tan in “An attack on Zarankiewicz’s problem through SAT solving” (2022). The lexicographic clause encoding is textbook (Knuth, TAOCP 7.2.2.2; and Codish and collaborators). The only new things in my version are the particular values that came out the other end.

That is, in a way, the point of the post. This is a standard technique that lives mostly inside the constraint-programming community and is underused everywhere else. If you find yourself staring at a SAT job that will not prove the impossibility you are sure is true, count the symmetries first. There may be a hundred trillion copies of your problem hiding in plain sight, and a few lexicographic clauses to make them go away.

Takeaways

• The UNSAT side of an exact-value SAT proof is the expensive half, and symmetry is usually why.
• Canonical-representative constraints (double-lex) are cheap, sound, and often decisive: they keep one grid per orbit instead of all m! times n! of them.
• They are standard in constraint programming and underused elsewhere.
• They scale on product-of-modest-groups symmetry (the Zarankiewicz grid) and stall on one large group (graph vertex relabeling), where canonical generation is the right tool instead. Match the method to the symmetry.

The code (a lexicographic-CNF builder, the Zarankiewicz attack, and the C4-free contrast case, all with tests) lives in my open-problems repository.

A. The Twelve Codes Together

Every post in this series answered a version of the same question: given a source of positive integers, how do you represent its values compactly as a sequence of bits? The answers differ in shape, in assumptions, and in which distribution each code implicitly expects.

Posts 1 and 2 (Kraft’s Inequality and McMillan’s Converse) established why prefix-free codes are the right unit of analysis. Post 3 (Universal Codes as Priors) named the frame: a code is a hypothesis about the source. Posts 4 through 10 filled in the catalogue. Posts 11 and 12 extended from integer coding to set representation, where the questions shift from “how long is this codeword?” to “how do you store membership?” and “how do you answer rank/select?”

Looking across all twelve, the main lesson is not that one code dominates. It is that the question “which code?” is always empirically answerable given a sample.

B. The Unifying Frame Restated

Post 3 introduced the codes-as-priors thesis with two instances behind it. We now have twelve. The thesis has not changed; it has only become more evidently true.

Here is the six-clause version.

Clause 1: a code is a hypothesis. Every prefix-free code for positive integers assigns a codeword length $\ell(n)$ to each value $n$. By Kraft’s equality for complete codes, the implied probability is $p(n) = 2^{-\ell(n)}$. A code is not neutral: it says something about how likely each value is.

Clause 2: lengths determine a prior. If you choose Elias gamma, you are implicitly betting that values are power-law distributed with exponent 2. If you choose Unary, you are betting on geometric(1/2). If you choose VByte, you are betting on a near-uniform distribution over byte-aligned ranges. The bet is in the lengths, whether or not you named it as a bet.

Clause 3: choosing a code is choosing a prior. This is just clauses 1 and 2 combined, but it bears stating as a unit: the act of picking a codec is a modeling decision, not an engineering neutral choice. Unary and Delta will produce radically different wire sizes on the same data, because they assume radically different sources.

Clause 4: the best code matches the actual source. The code that minimizes expected description length for a source $p$ is the one whose implied prior most closely matches $p$. Formally, expected length under code $c$ for source $p$ equals entropy $H(p)$ plus the KL divergence $D_\text{KL}(p ,|, q_c)$ where $q_c$ is the code’s implied prior. Minimum expected length occurs at $D_\text{KL} = 0$, i.e., when the code and source agree.

Clause 5: universal codes give bounded redundancy when the source is unknown. Elias gamma and delta are universal: for any source with a finite entropy upper-bound, their redundancy per symbol is $O(\log \log n)$ relative to the optimal code for that source. You pay a small overhead to avoid committing to a specific prior. The overhead is worth paying when you cannot measure the source in advance.

Clause 6: polyalgorithms adapt per-chunk rather than committing globally. RoaringBitmap is the clearest instance of this: it measures each 65536-element chunk and picks the container type (sorted array, bitset, run-length) that minimizes storage for that chunk’s observed density. This is not “pick the best universal code” but “pick the best structure from a menu, per block.” The principle generalizes: any large dataset may be non-stationary, and a single global codec will be suboptimal for every region where the source differs from the global average.

C. Composition with Type Algebra

Two posts in the companion Stepanov series made claims that interlock with everything above. Working through them changed how I think about codec design.

The post Bits Follow Types argued that codecs compose along the algebraic structure of types: a codec for Either<A, B> is built from codecs for A and B by prepending a tag bit and dispatching; a codec for Vec<A> is built from a codec for A by iterating. The composition law is structural. It follows the constructors of the algebraic data type, not the distribution of the data.

The post When Lists Become Bits argued that prefix-freeness lifts the free-monoid construction into bit space. Concatenating prefix-free codewords is unambiguous because each codeword is self-delimiting. The monoid of bit strings decomposes along codec boundaries exactly as the monoid of abstract tokens decomposes along type boundaries.

This series adds a third claim: the choice of leaf codec in any composite type is determined by the prior over that leaf’s data, while the composition structure is determined by the type. These are two orthogonal axes of design freedom.

Consider Either<uint32_t, std::string>. The composition structure says: write one bit for the variant tag, then write the payload for the selected variant. That part is fixed by the type. What is not fixed is which codec you use for the tag bit, and which codec you use for the uint32_t payload.

If 90% of values are uint32_t and 10% are std::string, a Huffman code over the two-symbol alphabet {left, right} gives you the tag bit nearly for free (0.47 bits expected vs. 1 bit for the flat tag). And if the uint32_t values cluster in [0, 127], VByte costs exactly 8 bits per value. Swap those observations and you swap those choices.

The type algebra determines the composition; the empirical distribution determines the leaf codecs. Neither axis constrains the other. Attending to both is what makes codec design non-trivial.

D. The Codec-Selection Library

The synthesis.hpp library distills the selection process into three functions.

empirical_distribution counts occurrences of each value in a sample and normalizes by the total count to produce a probability map.

redundancy_for computes the redundancy of a named code on a given distribution: expected length minus entropy. By Shannon’s source-coding theorem, this is always non-negative. A lower redundancy means the code’s implied prior is a closer fit to the actual source.

recommend_code ties them together: it estimates the empirical distribution from a sample, computes the redundancy of all six candidate universal codes (Unary, Gamma, Delta, Omega, Fibonacci, VByte), and returns the name of the code with the smallest redundancy.

inline std::string recommend_code(const std::vector<std::uint64_t>& sample)
{
    auto dist = empirical_distribution(sample);
    constexpr std::string_view candidates[] = {
        "Unary", "Gamma", "Delta", "Omega", "Fibonacci", "VByte"
    };
    double best_redundancy = std::numeric_limits<double>::infinity();
    std::string best_code;
    for (std::string_view candidate : candidates) {
        double r = redundancy_for(candidate, dist);
        if (r < best_redundancy) {
            best_redundancy = r;
            best_code = std::string(candidate);
        }
    }
    return best_code;
}

On a strongly geometric sample (value 1 appears much more often than value 2, which appears more often than value 3, and so on), recommend_code returns "Unary" or "Fibonacci". Both codes are matched to geometric-like priors; their redundancy on geometric data is low. On a sample of values in [500, 1000], it returns "VByte" or "Delta": byte-aligned codes have low overhead for values that cluster in that range, and neither Unary nor Gamma would be tolerable there. Unary would require hundreds of bits per value.

The point is not that recommend_code is a production-quality selector. It is that it makes the selection process concrete and mechanical. There is no best code in general. There is a best code given a sample.

E. The Six Principles

These are the things I now believe about coding, after working through twelve instances.

1. A code is a prior. Every codeword length implies a probability. Choosing a code is choosing what you believe about the source. The choice is never neutral, even when it is unconscious. When you reach for the first codec you know, you are betting on a prior you may never have examined.

2. Universality is robustness. A universal code performs well across many priors, not just one. Elias gamma is asymptotically optimal for any source from a broad class of power-law distributions; VByte is a pragmatic universal for byte-oriented hardware. Use universal codes when you do not know the prior in advance, when the prior may shift over time, or when the engineering cost of measuring the source outweighs the compression gain. Use Huffman or arithmetic coding when you do know the prior and the gain from exploiting it justifies the overhead.

3. Optimality is measurable. Shannon’s source-coding theorem gives the lower bound: no uniquely-decodable code can have expected length below the entropy $H(p)$ of the source. Every code’s redundancy is then measurable as expected-length minus entropy. This is not a theoretical abstraction; you can compute it from a sample in a few lines of code, as redundancy_for demonstrates. Pick codes by minimizing redundancy on the actual source, not by intuition about which code sounds right for the problem.

4. Engineering trades dominate at scale. Theoretical optima (Elias gamma, delta, arithmetic coding) are outperformed in throughput benchmarks by byte-aligned approximations (VByte) when decode speed is the binding constraint. The reason is cache line alignment and SIMD: modern CPUs process data in 16-byte to 64-byte chunks, and a code that crosses byte boundaries pays a penalty that no redundancy saving can overcome at typical data volumes. Figure out where the binding constraint lives (compression ratio, decode throughput, encode throughput, memory, latency) before you decide which axis to optimize.

5. Polyalgorithms beat single algorithms. When source characteristics vary across the data, adapt per-chunk rather than committing to a single global code. RoaringBitmap does this for integer sets: it observes each 65536-element chunk and picks the container type that minimizes storage for that chunk’s density. Zstd does this for byte streams: it switches between LZ77, Huffman, and ANS depending on local symbol statistics. The principle is general. Non-stationarity is the norm in real data, and a globally optimal code is locally suboptimal everywhere the source deviates from the global average.

6. The algebra of composition is orthogonal to the choice of leaf code. Type structure dictates how codecs compose: Either needs a tag, Vec needs a count or terminator, Product concatenates. These are structural facts about the algebraic data type and they do not change with the data distribution. What does change with the distribution is which codec you assign to each leaf. You can swap leaf codecs without changing the composition law, and you can change the composition structure without touching the leaf codecs. Keeping these two axes separate is what makes a codec library composable rather than a pile of special cases.

F. What Comes Next

This series covered prefix-free codes from their theoretical foundations through twelve instances, ending at polyalgorithms. I stopped here because this is where the foundations end and the engineering starts getting domain-specific. The frontier extends in three directions I have not covered.

Context mixing. The PAQ and ZPAQ family of compressors achieve near-arithmetic-limits on general data by running many predictive models in parallel and mixing their probability estimates with weights that are themselves adapted to the data. The key insight is that no single model is best everywhere; a mixture that weights models by their recent prediction accuracy outperforms any single model on non-stationary sources. Context mixing is the logical culmination of the codes-as-priors idea: you do not pick one prior, you maintain a portfolio.

Asymmetric Numeral Systems (ANS). Developed by Jaroslaw Duda around 2014, ANS achieves the compression ratio of arithmetic coding while decoding at 5 to 10 times the speed. It underlies LZ4’s entropy backend (FSE), Zstd’s entropy coder, Apple’s LZFSE, and several video compression standards. ANS works by maintaining a single integer state that encodes the accumulated probability of the symbols seen so far, updated with each new symbol using a lookup table rather than interval arithmetic. If you want to understand how modern fast compressors actually work at the symbol level, ANS is the answer.

Rate-distortion theory. Everything in this series assumed lossless compression: the decoder must recover the exact original bit sequence. Rate-distortion theory introduces a different class of trade, where you bound the allowable reconstruction error and ask how few bits you can use subject to that bound. This is the foundation of audio (MP3, AAC, Opus), video (H.264, H.265, AV1), and image compression (JPEG, WebP). The information-theoretic results here are just as clean as Shannon’s source-coding theorem, but the engineering is richer because the distortion measure (perceptual quality for audio and video, pixel error for images) shapes every design decision.

The principles from this series carry forward into all three directions: a code is a prior, universality is robustness, optimality is measurable, engineering trades dominate at scale. The new settings add new structure (context histories, state machines, distortion metrics) but not new foundations.

G. Cross-references

Posts in this series (in order): Kraft’s Inequality, McMillan’s Converse, Universal Codes as Priors, Unary and Elias Gamma, Elias Delta and Omega, Fibonacci Coding, Rice / Golomb, VByte / Varint, Huffman, Arithmetic Coding, Succinct Bit Vectors, RoaringBitmap.

Cross-series: Bits Follow Types and When Lists Become Bits.

PFC footnote: The production implementation of all 12 codes (and several more) is at github.com/queelius/wire-formats/tree/master/lib/pfc. This series develops the theory; PFC is the practice.

Sign in with GitHub to play. The “type a code at github.com/login/device” flow runs entirely in your browser; no server holds your token.

You can try it at /ask. There is a slider for how strongly the n-gram bleeds in.

I expected a small chatbot that sounds like me. I got a 135M parameter model that uses my words to produce paranoid lorem-ipsum, and a 1.7B model that mostly behaves like a competent chatbot decorated with my function words. Sometimes a phrase comes out that I might actually write. More often the output is grammatically OK but conceptually empty.

The result is weak. The architecture is interesting anyway, and I want to write up why.

The Pieces

Three things, all running locally:

1. SmolLM2-Instruct in 135M, 360M, or 1.7B sizes. Q4_K_M GGUF, served from HuggingFace, run via Wllama. 90 MB to 1 GB on disk. CPU only.

2. A token-level n-gram over my blog corpus: every post tokenized with SmolLM2’s BPE, indexed with a suffix array. 1.6 MB of source text, 470,000 tokens, 1.9 MB suffix array.

3. A token-by-token sampling loop that mixes the LLM’s output distribution with the n-gram’s, in probability space.

The third piece is the part worth thinking about.

The Math

At each generation step the LLM produces p_llm(t) over its 49,152-token vocabulary, exposed by Wllama’s getLogits(-1). The n-gram, given the longest suffix of the current context that occurs in the corpus, produces a sparse p_ngram(t): nonzero on tokens it has seen following that context, zero elsewhere.

Linear combination:

p_mix(t) = α · p_ngram(t) + (1 − α) · p_llm(t)

That is the whole algorithm. The inner loop is small enough to fit on screen:

for (let step = 0; step < N; step++) {
  const llm = await wllama.getLogits(-1);
  const m = ig.longestSuffixMatch(context);
  const ngram = m.suffixLen > 0
    ? new Map(ig.continuations(m.matchedTokens).map(c => [c.token, c.prob]))
    : null;

  const mix = new Map();
  for (const { token, p } of llm) {
    const pn = ngram?.get(token) ?? 0;
    mix.set(token, alpha * pn + (1 - alpha) * p);
  }

  const next = sample(mix, temperature);
  if (await wllama.isTokenEOG(next)) break;
  context.push(next);
  await wllama.decode([next]);
}

Tokens unseen by the n-gram have p_ngram = 0 and retain (1 − α) · p_llm in the mixture. They are not zeroed out, just unboosted.

α = 0 is the LLM. α = 1 is the n-gram, which loops as soon as the generated context drifts off-corpus. In between is a model running on LLM grammar with n-gram register.

What It Actually Produces

The 135M, prompt “Hello”, α = 0.1:

Hi! I’m working on a project with you! I’m trying to hide some of your work. I’ll have to wait for your response, but I’ll give you a brief summary to let you know what I’m working on. Hope you’re doing well!

The vocabulary is mine. I write about hiding data, encrypted search, comparing approaches. The 135M has no idea what those concepts mean. It glues my words into paragraphs that read like a paranoid academic argument with the visitor.

The 1.7B is better but the failure mode is more subtle. With α = 0.1 it produces grammatical paragraphs that drop in occasional Alex-shaped phrases (“compositional depth”, “rules as data”, project names from my corpus). At α = 0 it sounds like a generic chatbot. At α = 1 it sounds like me, in fragments, looping.

What mixing-at-sample-time actually does, I think, is pull register without pulling competence. The n-gram makes the surface me-shaped. It does not give the model any of the conceptual machinery that lets me mean something when I use those words. The output is texture, not thought.

Why The Architecture Is Interesting Anyway

The result is weak, but the structural properties are real:

The corpus stays uncompressed. Conventional fine-tuning is a lossy gradient encoding of your training distribution. Here, the corpus ships in full. The “fine-tune” is a static asset you can read.

Adding a document is a 30 ms rebuild. Not a training run.

Removing a document is the same operation reversed. No question of whether the model “forgot” it. There is no gradient that could remember.

Auditing memorization is a binary search. O(log n) to find every occurrence of a span in the corpus. With gradient-descent fine-tuning, you cannot easily answer this.

Tokenizer parity makes model swaps free. All three SmolLM2 sizes share the BPE, so the same suffix array works against any of them.

These properties are why I think the idea has legs even though the demo doesn’t.

What’s Wrong With It

The failure modes I observed, mostly so I know what to fix:

Linear mixing is a soft prior, not a guide. It lifts tokens. It does not constrain meaning. The model still hallucinates; the prior just makes it hallucinate in my register.

Token-level matching is too local. The n-gram looks at the longest in-corpus suffix, which is usually a handful of tokens. Enough for register, not enough for thematic coherence. Sequential sentences can pull from completely different parts of the corpus.

The corpus is too small. 1.6 MB has only so many patterns. With α near 1 the model loops within seconds.

Tiny models cannot be steered into competence. The 135M under any prior is a syntax engine. With the prior it is a syntax engine with my vocabulary. Useful for a curio, not for a tool.

Temperature interacts unpredictably. Applying temperature to the mixed distribution is mathematically clean, but small changes to T produce large changes in output character that I cannot tune intuitively.

What Would Make It Work

Roughly in order of how much I think each would help:

Passage-level retrieval, also mixed in. A second prior at the level of full passages, retrieved via embedding search, with passage-level distributions mixed alongside the token-level n-gram. Token-level for register, passage-level for theme.

Logit-space mixing instead of probability-space. log p_mix = log p_llm + λ · log(p_ngram + ε) actively penalizes tokens with low corpus support, rather than only adding lift to high-support ones. Sharper steering.

A different LLM family. SmolLM2 caps at 1.7B. Qwen 2.5 1.5B or Llama 3.2 3B are similar size and stronger on instruction-following. Requires retokenizing the corpus; the JS doesn’t change.

Context-dependent α. Right now α is fixed. When the longest match is short (the LLM is off-corpus), trust the LLM more. When the match is long (the LLM is producing in-corpus text), trust the prior more. Probably the change with the most upside of these.

A Note On Local-First

The reason the whole pipeline runs in the browser, even though the result is weak, is that this is the use case fine-tuning has historically been worst at: people who want a personal model from their own writing, without running a GPU cluster, without uploading their writing anywhere, and without ending up with an opaque-weights artifact they cannot inspect.

Sample-time mixing with a public LLM is one possible answer. The model is a public utility, the corpus is a static asset, the mixing is in user space. Anyone curious about what is being applied to the model can read the code.

This particular implementation is mediocre. The shape of the answer is right.

What This Is

Not a fine-tune. Not a chatbot that sounds like me. A research artifact. A thing that shows LLM and n-gram can be composed at sample time, with all the structural properties that follow, and that the architecture is worth more work even though this particular composition does not yet land.

The toy is at /ask and /infinigram. Have at it.