The idea

Some problems are easy in one direction and hard in the other. Taking a derivative is mechanical. Finding an antiderivative can require genuine creativity. Generating a random expression and verifying a proof is easy. Discovering the proof is hard.

RPSDG (Reverse-Process Synthetic Data Generation) exploits this asymmetry. Run the easy direction with full step-by-step work, then reverse the result to get a hard problem with a known solution. You end up with process-supervised training data: not just the answer, but the entire derivation.

Richard Sutton’s “The Bitter Lesson” argues that methods scaling with compute and data will eventually win. The bottleneck is high-quality data. A lot of the world’s data is latent, the processes that generated it are not written down. In math, the way a proof was discovered is usually hidden behind a polished presentation. RPSDG is one way to manufacture that hidden process data.

Derivatives to Integrals

Computing derivatives is mechanical. Integration often is not. That asymmetry gives us a data pipeline.

1. Start with known functions. Pick functions \( f(x) \) with closed-form derivatives: polynomials, trig, exponentials, logarithms. Vary complexity.

2. Differentiate with full work shown. Take the derivative of \( f(x) \) to get \( f'(x) \), recording every step.

3. Reverse the process. Now \( f'(x) \) is the problem and \( f(x) \) is the solution. The recorded steps, read backward, give you a worked integration example.

By composing functions of varying complexity, you get integration problems of graduating difficulty. The training data comes with step-by-step solutions for free, because you generated it by running the easy direction.

Proofs by Random Walk

The same idea works for theorem proving. Generating proofs is hard. Verifying them is (comparatively) easy.

1. Random walks in expression space. Start with a random expression \( e_{\text{start}} \). Apply rewrite rules \( r_1, r_2, \ldots, r_n \) to get a chain of intermediate expressions ending at \( e_{\text{end}} \).

2. Read off the theorem. The pair \( (e_{\text{start}}, e_{\text{end}}) \) is a theorem. The chain of rewrites is its proof.

3. Reverse when useful. Running the chain backward works too, especially when a complex step in one direction (integration) becomes simple in the other (differentiation).

4. Scale it. Random starting points and random rewrite sequences give you a diverse set of theorems and proofs automatically. No human has to come up with the theorem first.

What this gets you

The training data has process supervision baked in. Every example includes intermediate steps, not just the final answer. That should help LLMs learn multi-step reasoning rather than pattern-matching to answers.

It also gives you something like explainability for free: the model’s training data literally consists of step-by-step solutions, so the model has a better chance of producing step-by-step reasoning at inference time.

Limitations and next steps

This is early work. The data generation pipeline exists, but I have not yet run the full fine-tuning experiments and benchmarks. The planned pipeline is:

• Data generation with graduating difficulty (curriculum learning)
• Fine-tuning transformer-based LMs on the generated data
• Self-supervised learning experiments
• Evaluation against standard math reasoning benchmarks

Further out, I want to explore reinforcement learning for rewarding multi-step reasoning even when the solution is not known in advance but can be verified.

I demoed two projects:

1. A simple Colab notebook using basic Python to generate text with an n-gram model. The point was to illustrate the core idea behind language models and show concretely why n-grams fall short, which motivates the transformer architecture.

2. A project that uses ElasticSearch and LLMs to allow natural language search queries over databases.

The talk went well. Content and code are on GitHub.

The Problem

In reliability engineering, you often find yourself in an annoying situation: a system fails, but you do not know which component caused the failure. This is called masked failure data. On top of that, some systems are still running when you stop observing them, so you only know they survived at least that long. That is right censoring. Both are common in practice. Identifying the exact failed component is expensive or sometimes impossible.

The project builds a likelihood-based framework that handles both masking and censoring simultaneously, models component lifetimes with Weibull distributions, derives closed-form Fisher information for the exponential special case, and provides bootstrap methods for uncertainty quantification. I implemented it all in an R package so practitioners can actually use it.

Related Work

This connects to several other posts and projects:

• Closed-Form Results for Masked Exponential Series Systems covers the exponential distribution special case with analytical solutions
• likelihood.model R package is the software implementation

See the full project page here.

This seemingly simple idea has profound implications. In this post, I’ll show how to build a content-addressed virtual filesystem in Python, exploring how immutability and content addressing work together to create elegant, functional data structures.

Why Content Addressing?

Consider a traditional filesystem. When you modify a file, the system overwrites the old content. The file’s identity (its path) stays the same, but its content changes. This mutable approach has problems:

1. No automatic history: Once you overwrite, the old data is gone
2. No deduplication: Two identical files take up twice the space
3. No integrity verification: Corruption can go undetected

Content addressing solves all three. If a file’s identity is its content (via a hash), then:

1. History is preserved: Changing content creates a new hash, so the old version still exists
2. Deduplication is automatic: Identical content has identical hashes—stored once
3. Integrity is built-in: If the content doesn’t match the hash, you know something’s wrong

The Node Hierarchy

Let’s build this. First, we define our filesystem nodes using Python’s frozen dataclasses:

from dataclasses import dataclass, field
import hashlib
import json

@dataclass(frozen=True)
class Node:
    """Base class for all filesystem nodes."""
    mode: int
    uid: int = 1000
    gid: int = 1000
    mtime: float = field(default_factory=time.time)

    def compute_hash(self) -> str:
        """Compute SHA256 hash of this node including all metadata."""
        data = json.dumps(self.to_dict(), sort_keys=True)
        return hashlib.sha256(data.encode()).hexdigest()

The frozen=True parameter is crucial. It makes instances immutable—you cannot modify a Node after creation. Any “change” requires creating a new Node.

We then specialize for different node types:

@dataclass(frozen=True)
class FileNode(Node):
    """Regular file node."""
    content: bytes = b""

@dataclass(frozen=True)
class DirNode(Node):
    """Directory node containing references to child nodes."""
    children: Dict[str, str] = field(default_factory=dict)  # name -> hash

Notice that DirNode.children maps names to hashes, not to Node objects directly. This is the key insight: directories don’t contain files; they contain references to file hashes. The actual nodes live in a separate store.

The DAG Structure

This reference-based approach creates a Directed Acyclic Graph (DAG):

class FileSystem:
    """Content-addressable virtual filesystem."""

    def __init__(self):
        # The DAG: hash -> Node
        self.nodes: Dict[str, Node] = {}

        # Path index: absolute path -> hash
        self.paths: Dict[str, str] = {}

    def _add_node(self, node: Node) -> str:
        """Add a node to the DAG, returning its hash."""
        node_hash = node.compute_hash()
        if node_hash not in self.nodes:
            self.nodes[node_hash] = node
        return node_hash

When we add a node, we compute its hash and store the mapping hash → node. If an identical node already exists (same hash), we don’t duplicate it—we just return the existing hash. Deduplication is automatic.

Immutable Updates

Here’s where immutability shines. When we write to a file, we don’t modify anything. Instead, we:

1. Create a new FileNode with the new content
2. Create a new DirNode for the parent, pointing to the new file hash
3. Update the path index

def write(self, path: str, content: bytes) -> bool:
    """Write content to a file."""
    parent_path, name = self._get_parent_path(path)
    parent_hash = self.paths[parent_path]
    parent = self.nodes[parent_hash]

    # Create new file node
    file_node = FileNode(content)
    file_hash = self._add_node(file_node)

    # Create new parent directory with updated child reference
    new_children = dict(parent.children)
    new_children[name] = file_hash
    new_parent = DirNode(children=new_children)
    new_parent_hash = self._add_node(new_parent)

    # Update path index
    self.paths[parent_path] = new_parent_hash
    self.paths[path] = file_hash

    return True

The old FileNode still exists in self.nodes. The old DirNode still exists too. We’ve just created new versions and updated where the path points. This is structural sharing—unchanged parts of the tree are shared between versions.

Visualizing the DAG

Let’s trace through an example:

fs = FileSystem()
fs.mkdir("/project")
fs.write("/project/main.py", b"print('hello')")
fs.write("/project/main.py", b"print('world')")

After these operations, our DAG contains:

Hash: a1b2c3... → DirNode(children={})           # original /project
Hash: d4e5f6... → FileNode("print('hello')")     # first version
Hash: g7h8i9... → DirNode(children={"main.py": "d4e5f6..."})
Hash: j0k1l2... → FileNode("print('world')")     # second version
Hash: m3n4o5... → DirNode(children={"main.py": "j0k1l2..."})

Both versions of main.py exist. The path /project/main.py points to the latest hash (j0k1l2...), but we could easily restore the old version if we tracked which hashes corresponded to which versions.

Benefits in Practice

This design enables powerful features almost for free:

Snapshots: Save the current paths dictionary. Restore it later to go back in time.

def snapshot(self) -> Dict[str, str]:
    """Create a snapshot of the current filesystem state."""
    return dict(self.paths)

def restore(self, snapshot: Dict[str, str]):
    """Restore filesystem to a previous snapshot."""
    self.paths = dict(snapshot)

Deduplication: Multiple paths can point to the same hash.

# These might share the same underlying node if content is identical
fs.write("/file1.txt", b"hello")
fs.write("/file2.txt", b"hello")  # Same hash, no new storage

Integrity checking: If someone asks for a file, we can verify it.

def verify(self, path: str) -> bool:
    """Verify a file's integrity."""
    node_hash = self.paths[path]
    node = self.nodes[node_hash]
    return node.compute_hash() == node_hash

The Functional Programming Connection

This approach is deeply connected to functional programming. In FP:

• Data is immutable
• “Changes” create new values
• Sharing is safe because nothing mutates

Our filesystem follows these principles exactly. Nodes are frozen. “Writing” creates new nodes. Multiple paths can safely share nodes because nodes never change.

This is why Clojure’s persistent data structures, Haskell’s pure values, and Git’s object store all use similar ideas. Content addressing + immutability = safe, efficient, verifiable data.

Trade-offs

Nothing is free. This approach has costs:

1. Memory: Old versions accumulate. You need garbage collection to reclaim space from unreachable nodes.

2. Performance: Creating new nodes for every change can be slower than in-place mutation for write-heavy workloads.

3. Complexity: Path resolution requires extra indirection through the hash table.

For many use cases—especially those valuing history, integrity, and safe concurrency—these trade-offs are worthwhile.

Conclusion

Content addressing transforms how we think about data. Instead of “where is this file?” we ask “what is this content’s identity?” Instead of destructive updates, we create new versions while sharing unchanged structure.

This pattern appears everywhere: Git, IPFS, Nix, Docker layers, and many database internals. Understanding it opens doors to building robust, elegant systems.

The full implementation in DagShell extends these ideas with a complete POSIX-like interface, demonstrating how content addressing can underpin a full virtual filesystem.

Next in this series: Unix Philosophy in Python — building composable commands with method chaining.

The idea: take a task that large models handle well (translating “find all orders over $100 from last month” into the corresponding ElasticSearch JSON query) and see if a tiny model can learn it from synthetic data.

The data pipeline works like this. I started by generating examples from GPT-4. Then I wrote a script that samples from those outputs and uses them as few-shot examples for Mistral, which generates a much larger synthetic dataset. The next step is reshaping the data into the expected format and fine-tuning.

This is early stage. The synthetic data exists, the fine-tuning has not happened yet. I will update this post with results.

If you are interested in collaborating on this or related projects, email me at lex@metafunctor.com.

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

This philosophy gave us pipes (|), small focused utilities (cat, grep, sort), and the ability to compose complex operations from simple parts:

cat access.log | grep "404" | cut -d' ' -f1 | sort | uniq -c | sort -rn

Can we bring this composability to Python? Not just subprocess calls, but native Python objects that pipe and chain as naturally as Unix commands? Let’s build it.

The Problem with Methods

Standard Python methods have a composition problem. Consider:

# We want to: read a file, filter lines, count words
content = read_file("/data/log.txt")
filtered = grep(content, "ERROR")
count = wc(filtered)

This works, but it’s inside-out compared to how we think about it. We think “read, then filter, then count”—a left-to-right pipeline. But we write it bottom-up with intermediate variables.

Method chaining helps:

read_file("/data/log.txt").grep("ERROR").wc()

Now it flows left-to-right. But how do we make arbitrary operations chainable?

The CommandResult Pattern

The key insight is wrapping every result in a chainable container:

@dataclass
class CommandResult:
    """Wrapper that enables method chaining."""
    data: Any           # The actual result data
    text: str = None    # Text representation
    exit_code: int = 0  # Unix-style exit code
    _shell: 'DagShell' = None  # Reference back to the shell

    def __str__(self) -> str:
        if self.text is not None:
            return self.text
        return str(self.data)

Every command returns a CommandResult. The result carries the data and a reference to the shell that produced it. This reference enables chaining—we can call more methods through it.

Making Commands Chainable

Here’s a simple command implementation:

class DagShell:
    def echo(self, *args) -> CommandResult:
        """Echo arguments to output."""
        text = ' '.join(str(arg) for arg in args)
        return CommandResult(
            data=text,
            text=text,
            exit_code=0,
            _shell=self
        )

The magic happens in CommandResult. We add methods that delegate back to the shell:

@dataclass
class CommandResult:
    # ... fields as before ...

    def grep(self, pattern: str) -> 'CommandResult':
        """Filter lines matching pattern."""
        return self._shell.grep(pattern, input_data=self.data)

    def wc(self, *flags) -> 'CommandResult':
        """Count lines, words, or characters."""
        return self._shell.wc(*flags, input_data=self.data)

    def out(self, path: str) -> 'CommandResult':
        """Redirect output to a file."""
        self._shell.fs.write(path, str(self).encode())
        return self

Now we can chain:

shell.echo("hello world").wc("-w")  # Returns: 2
shell.cat("/data/log.txt").grep("ERROR").wc("-l")  # Count error lines

The Dual Nature: Objects and Files

Unix commands have a superpower: the same output can go to the screen, a file, or another command. We can replicate this:

# Chain more commands
result = shell.cat("/data/log.txt").grep("ERROR")

# Get as Python object
lines = result.lines()  # List[str]

# Or redirect to a file
result.out("/data/errors.txt")

The CommandResult is simultaneously:

1. A Python object you can inspect and manipulate
2. A text stream you can redirect to files
3. An input source for the next command in a pipeline

This dual nature—structured data and text stream—bridges the gap between Unix philosophy and Python’s object orientation.

Implementing Piping

True Unix pipes pass data between processes. We simulate this with a “last result” mechanism:

class DagShell:
    def __init__(self):
        self._last_result: Optional[CommandResult] = None

    def _(self) -> CommandResult:
        """Return the last command's result (like $? or $_)."""
        if self._last_result is None:
            return CommandResult(data='', text='', exit_code=0, _shell=self)
        return self._last_result

Now we can build pipelines:

shell.cat("/data/log.txt")
shell._().grep("ERROR")
shell._().wc("-l")

Or more elegantly, commands can accept piped input:

def grep(self, pattern: str, input_data=None) -> CommandResult:
    """Filter lines matching pattern."""
    if input_data is None:
        input_data = self._last_result.data if self._last_result else ''

    lines = str(input_data).splitlines()
    matching = [line for line in lines if pattern in line]

    result = CommandResult(
        data=matching,
        text='\n'.join(matching),
        exit_code=0 if matching else 1,
        _shell=self
    )
    self._last_result = result
    return result

Method Chaining in Action

Let’s build something real—a log analysis pipeline:

# Create a shell and some test data
shell = DagShell()
shell.mkdir("/logs")
shell.echo("""
2024-01-15 10:30:00 INFO  User login: alice
2024-01-15 10:31:00 ERROR Database connection failed
2024-01-15 10:32:00 INFO  User login: bob
2024-01-15 10:33:00 ERROR Timeout waiting for response
2024-01-15 10:34:00 WARN  High memory usage
2024-01-15 10:35:00 ERROR Disk space low
""".strip()).out("/logs/app.log")

# Pipeline: find errors, extract timestamps, save to file
(shell
    .cat("/logs/app.log")
    .grep("ERROR")
    .cut(delimiter=" ", fields="1,2")
    .out("/logs/error_times.txt"))

# Read the result
print(shell.cat("/logs/error_times.txt"))
# Output:
# 2024-01-15 10:31:00
# 2024-01-15 10:33:00
# 2024-01-15 10:35:00

Each step does one thing. The chain composes them into a useful operation. The data flows left-to-right, just like our mental model.

Directory Navigation with a Stack

Unix has cd, but shells also have pushd and popd for directory stacks. We implement this:

class DagShell:
    def __init__(self):
        self._cwd = '/'
        self._dir_stack = []

    def pushd(self, path: str) -> CommandResult:
        """Push current directory and change to new one."""
        self._dir_stack.append(self._cwd)
        return self.cd(path)

    def popd(self) -> CommandResult:
        """Pop directory from stack and change to it."""
        if not self._dir_stack:
            return CommandResult(data='', text='popd: directory stack empty', exit_code=1)
        old_dir = self._dir_stack.pop()
        return self.cd(old_dir)

Now we can navigate without losing our place:

shell.pushd("/project/src")
# ... work in src ...
shell.pushd("tests")
# ... work in tests ...
shell.popd()  # back to /project/src
shell.popd()  # back to original directory

Exit Codes: Success and Failure

Unix commands return exit codes: 0 for success, non-zero for failure. We include this in CommandResult:

def grep(self, pattern: str) -> CommandResult:
    # ... filtering logic ...
    return CommandResult(
        data=matching,
        text='\n'.join(matching),
        exit_code=0 if matching else 1,  # 1 if no matches
        _shell=self
    )

This enables conditional logic:

result = shell.grep("pattern", file="/data/log.txt")
if result.exit_code == 0:
    print(f"Found {len(result.lines())} matches")
else:
    print("No matches found")

The Philosophy Applied

Let’s revisit McIlroy’s principles and see how we’ve applied them:

“Do one thing well”: Each method (cat, grep, wc, cut) does exactly one thing.

“Work together”: CommandResult enables any command to connect to any other.

“Text streams as universal interface”: Every result has a text representation via __str__, making it redirectable and pipeable.

We’ve also added Python-specific benefits:

• Type safety: Results carry structured data, not just text
• Introspection: result.lines(), result.data, etc.
• Chaining: Method chains read left-to-right like pipelines

The Fluent Pattern

This is an instance of the Fluent Interface pattern, where methods return self (or a related object) to enable chaining. Martin Fowler described it in 2005, but the idea is older—Smalltalk embraced it from the beginning.

The key is designing methods that:

1. Perform their action
2. Return something chainable
3. Maintain enough context for the next operation

When done well, code reads almost like prose:

(shell
    .mkdir("/project")
    .cd("/project")
    .echo("# My Project")
    .out("README.md")
    .echo("def main(): pass")
    .out("main.py")
    .ls("-la"))

Trade-offs

This approach isn’t free:

1. Wrapper overhead: Every result is wrapped in CommandResult
2. Learning curve: Users must understand the chaining pattern
3. Debugging: Long chains can be hard to debug—where did it fail?

For scripting and exploratory work, the expressiveness outweighs these costs. For performance-critical code, you might unwrap to raw operations.

Conclusion

The Unix philosophy isn’t about Unix—it’s about composability. Small, focused operations that connect through a universal interface create systems greater than the sum of their parts.

In Python, we achieve this with:

• Wrapper types that carry data and context
• Method chaining for left-to-right flow
• Dual representations as objects and text streams
• Exit codes for success/failure signaling

The result is a fluent interface where complex operations emerge from simple, composable parts—Unix philosophy, realized in Python.

See the full implementation in DagShell, which builds a complete virtual filesystem with this fluent pattern.

Next in this series: Embedding a Scheme Interpreter — adding a DSL for filesystem scripting.

But embedding a full language runtime is heavy. What if we just need something small—variables, functions, conditionals, loops? A Lisp-family language is perfect for this: the syntax is trivial to parse, the semantics are clean, and it’s powerful enough for real scripting.

In this post, I’ll walk through building a minimal Scheme interpreter in Python and integrating it with a virtual filesystem. We’ll cover the complete pipeline: tokenization, parsing, evaluation, and extension with custom primitives.

Why Scheme?

Scheme is a minimalist Lisp dialect. Its syntax is just parentheses and atoms:

(define x 10)
(+ x (* 2 3))
(if (> x 5) "big" "small")

This uniformity makes parsing trivial. There’s no operator precedence, no complex grammar—just:

1. Atoms: numbers, strings, symbols
2. Lists: (thing thing thing ...)

That’s it. The entire parser can be written in under 50 lines.

Step 1: Tokenization

First, we convert source code into tokens. Scheme’s tokenization is simple: split on whitespace, but respect strings and parentheses.

def tokenize(text: str) -> List[str]:
    """Convert Scheme code into tokens."""
    # Handle comments (lines starting with ;)
    lines = text.split('\n')
    text = '\n'.join(line.split(';')[0] for line in lines)

    # Add spaces around parens for easy splitting
    text = text.replace('(', ' ( ').replace(')', ' ) ')

    # Split, handling string literals
    tokens = []
    in_string = False
    current = []

    for char in text:
        if char == '"':
            in_string = not in_string
            current.append(char)
        elif in_string:
            current.append(char)
        elif char.isspace():
            if current:
                tokens.append(''.join(current))
                current = []
        else:
            current.append(char)

    if current:
        tokens.append(''.join(current))

    return tokens

Input: (define x (+ 1 2)) Output: ['(', 'define', 'x', '(', '+', '1', '2', ')', ')']

Step 2: Parsing

Parsing converts tokens into an Abstract Syntax Tree (AST). In Scheme, the AST is just nested Python lists:

@dataclass
class Symbol:
    """Represents a Scheme symbol."""
    name: str

def parse(tokens: List[str]) -> Any:
    """Parse tokens into an AST."""
    def parse_expr(index: int) -> Tuple[Any, int]:
        token = tokens[index]

        if token == '(':
            # Parse a list
            lst = []
            index += 1
            while tokens[index] != ')':
                expr, index = parse_expr(index)
                lst.append(expr)
            return lst, index + 1

        elif token == ')':
            raise SyntaxError("Unexpected )")

        else:
            # Parse an atom
            return parse_atom(token), index + 1

    expr, _ = parse_expr(0)
    return expr

def parse_atom(token: str) -> Any:
    """Parse a single atom."""
    # Try integer
    try:
        return int(token)
    except ValueError:
        pass

    # Try float
    try:
        return float(token)
    except ValueError:
        pass

    # String literal
    if token.startswith('"') and token.endswith('"'):
        return token[1:-1]

    # Boolean
    if token == '#t':
        return True
    if token == '#f':
        return False

    # Symbol
    return Symbol(token)

Input tokens: ['(', 'define', 'x', '(', '+', '1', '2', ')', ')'] Output AST: [Symbol('define'), Symbol('x'), [Symbol('+'), 1, 2]]

The beauty of Lisp: the AST is the syntax. There’s no separate tree structure—it’s just lists of symbols and values.

Step 3: The Environment

Before evaluation, we need an environment to track variable bindings. This is where lexical scoping lives:

class Environment:
    """Lexical environment for variable bindings."""

    def __init__(self, parent: Optional['Environment'] = None):
        self.bindings: Dict[str, Any] = {}
        self.parent = parent

    def define(self, name: str, value: Any):
        """Define a new binding in this environment."""
        self.bindings[name] = value

    def get(self, name: str) -> Any:
        """Look up a binding, checking parent scopes."""
        if name in self.bindings:
            return self.bindings[name]
        elif self.parent:
            return self.parent.get(name)
        else:
            raise NameError(f"Undefined variable: {name}")

    def set(self, name: str, value: Any):
        """Update an existing binding."""
        if name in self.bindings:
            self.bindings[name] = value
        elif self.parent:
            self.parent.set(name, value)
        else:
            raise NameError(f"Undefined variable: {name}")

Each function call creates a new Environment with the current one as its parent. This chain enables closures—inner functions that remember their enclosing scope.

Step 4: Evaluation

The evaluator is the heart of the interpreter. It walks the AST and computes values:

def evaluate(expr: Any, env: Environment) -> Any:
    """Evaluate an expression in an environment."""

    # Self-evaluating: numbers, strings, booleans
    if isinstance(expr, (int, float, str, bool, type(None))):
        return expr

    # Variable lookup
    if isinstance(expr, Symbol):
        return env.get(expr.name)

    # Must be a list (function call or special form)
    if not isinstance(expr, list) or not expr:
        return expr

    op = expr[0]

    # Special forms
    if isinstance(op, Symbol):
        if op.name == 'quote':
            return expr[1]

        if op.name == 'define':
            name = expr[1]
            value = evaluate(expr[2], env)
            env.define(name.name, value)
            return value

        if op.name == 'if':
            condition = evaluate(expr[1], env)
            if condition:
                return evaluate(expr[2], env)
            elif len(expr) > 3:
                return evaluate(expr[3], env)
            return None

        if op.name == 'lambda':
            params = expr[1]
            body = expr[2]
            return Procedure(params, body, env)

    # Function application
    func = evaluate(op, env)
    args = [evaluate(arg, env) for arg in expr[1:]]
    return func(*args)

Special forms (define, if, lambda, etc.) have custom evaluation rules. Everything else is a function call: evaluate the operator, evaluate the arguments, call the function.

Step 5: User-Defined Functions

The lambda form creates procedures. A Procedure captures its parameters, body, and defining environment:

@dataclass
class Procedure:
    """A user-defined function."""
    params: List[Symbol]
    body: Any
    env: Environment

    def __call__(self, *args):
        # Create a new environment for the call
        local_env = Environment(parent=self.env)

        # Bind parameters to arguments
        for param, arg in zip(self.params, args):
            local_env.define(param.name, arg)

        # Evaluate the body in this new environment
        return evaluate(self.body, local_env)

When called, a Procedure:

1. Creates a new environment with its defining environment as parent (closure!)
2. Binds parameters to the passed arguments
3. Evaluates its body in this new environment

This simple structure gives us closures, higher-order functions, and lexical scoping.

Step 6: Built-in Primitives

The global environment provides built-in functions:

def create_global_env() -> Environment:
    """Create environment with built-in primitives."""
    env = Environment()

    # Arithmetic
    env.define('+', lambda *args: sum(args))
    env.define('-', lambda a, b=None: -a if b is None else a - b)
    env.define('*', lambda *args: reduce(lambda x, y: x * y, args, 1))
    env.define('/', lambda a, b: a / b)

    # Comparison
    env.define('=', lambda a, b: a == b)
    env.define('<', lambda a, b: a < b)
    env.define('>', lambda a, b: a > b)

    # List operations
    env.define('car', lambda lst: lst[0])
    env.define('cdr', lambda lst: lst[1:])
    env.define('cons', lambda a, b: [a] + list(b))
    env.define('list', lambda *args: list(args))
    env.define('null?', lambda lst: lst == [])

    # Higher-order functions
    env.define('map', lambda f, lst: [f(x) for x in lst])
    env.define('filter', lambda f, lst: [x for x in lst if f(x)])
    env.define('reduce', lambda f, lst, init: reduce(f, lst, init))

    return env

Each primitive is just a Python function. The interpreter treats them identically to user-defined procedures.

Step 7: Filesystem Integration

Now the interesting part: extending Scheme with filesystem primitives.

def create_global_env(shell=None) -> Environment:
    env = Environment()

    # ... standard primitives ...

    # Filesystem primitives (if shell provided)
    if shell:
        env.define('ls', lambda path='/': shell.ls(path).lines())
        env.define('cat', lambda path: shell.cat(path).text)
        env.define('mkdir', lambda path: shell.mkdir(path) and path)
        env.define('write', lambda path, content:
            shell.fs.write(path, content.encode()) and path)
        env.define('exists?', lambda path: shell.fs.exists(path))
        env.define('cd', lambda path: shell.cd(path) and path)
        env.define('pwd', lambda: shell._cwd)

    return env

Now Scheme can script filesystem operations:

; Create a project structure
(mkdir "/project")
(mkdir "/project/src")
(mkdir "/project/tests")

(write "/project/README.md" "# My Project\n")
(write "/project/src/main.py" "def main(): pass\n")

; List files
(ls "/project")
; => ("README.md" "src" "tests")

; Check existence
(if (exists? "/project/README.md")
    "Found it!"
    "Not found")

Step 8: The REPL

A Read-Eval-Print Loop ties it together:

class SchemeREPL:
    def __init__(self, shell=None):
        self.env = create_global_env(shell=shell)

    def eval_string(self, code: str) -> Any:
        """Evaluate a string of Scheme code."""
        tokens = tokenize(code)
        if not tokens:
            return None

        result = None
        idx = 0

        while idx < len(tokens):
            expr, idx = self._parse_one(tokens, idx)
            result = evaluate(expr, self.env)

        return result

    def run(self):
        """Interactive REPL."""
        print("Scheme REPL (type 'quit' to exit)")
        while True:
            try:
                code = input("scheme> ")
                if code.strip() == 'quit':
                    break
                result = self.eval_string(code)
                if result is not None:
                    print(format_value(result))
            except Exception as e:
                print(f"Error: {e}")

Example Session

Here’s what a session looks like:

scheme> (define double (lambda (x) (* x 2)))
scheme> (double 21)
42

scheme> (map double (list 1 2 3 4 5))
(2 4 6 8 10)

scheme> (mkdir "/data")
"/data"

scheme> (write "/data/numbers.txt" "1\n2\n3\n")
"/data/numbers.txt"

scheme> (cat "/data/numbers.txt")
"1\n2\n3\n"

scheme> (filter (lambda (x) (> x 1))
               (map (lambda (s) (string->number s))
                    (string-split (cat "/data/numbers.txt") "\n")))
(2 3)

We’ve built a scriptable filesystem with a dozen lines of primitives!

Design Patterns

Several patterns make this work:

Homoiconicity: Code is data. The AST is just lists, so we can generate and manipulate code programmatically.

Closures: Functions capture their environment, enabling powerful patterns like partial application and callbacks.

Extension via primitives: Adding features is just adding functions to the global environment. No interpreter changes needed.

Separation of concerns: Tokenizing, parsing, and evaluating are distinct phases with clean interfaces.

Trade-offs

This simple interpreter has limitations:

1. No tail-call optimization: Deep recursion will overflow the Python stack.
2. No macros: We can’t extend the language’s syntax.
3. Error messages are basic: Line numbers would require more tracking.
4. Performance: Pure interpretation is slow for heavy computation.

For a DSL extending an application, these trade-offs are often acceptable. The simplicity enables understanding and modification.

Conclusion

Embedding a language isn’t black magic. At its core:

1. Tokenize: Break text into tokens
2. Parse: Build an AST (for Lisp, just nested lists)
3. Evaluate: Walk the AST, computing values
4. Extend: Add primitives that call into your application

Scheme’s minimal syntax makes this especially tractable. In a few hundred lines, we get variables, functions, closures, conditionals, recursion, and extensibility.

The result is a scriptable application where users can automate, experiment, and build beyond what the core interface provides—exactly what we want from a DSL.

The complete implementation is in DagShell, demonstrating Scheme as a filesystem scripting language alongside Python’s fluent API.

This is the final post in the series. Previous posts: Immutable Content-Addressed Filesystems and Unix Philosophy in Python.

That part was straightforward. The part that got interesting was highlighting: tracking which sentence is being spoken and lighting it up in the browser as the audio plays.

The Pipeline

Three commands:

narro hugo install ~/mysite    # copy player assets (JS, CSS, HTML partial)
narro hugo generate ~/mysite   # narrate all posts with tts: true
narro hugo status ~/mysite     # show what's been narrated and what hasn't

generate walks the content directory, finds posts with tts: true in frontmatter, extracts the prose (stripping code blocks, math, shortcodes, frontmatter), runs it through narro, converts to Opus via ffmpeg, and drops narration.opus and narration.json next to the index.md. The JSON file contains sentence-level timestamps. The HTML partial picks them up and wires everything together.

The player is a vanilla JS widget. No dependencies. Play, pause, spacebar toggle. The active paragraph highlights as the audio plays.

The Alignment Problem

Timestamps are the interesting part. Narro uses a causal language model (Qwen-based, 80M params) to generate hidden states, then a Vocos decoder converts those to audio. There is no explicit alignment signal in this architecture.

My first attempt tried word-level timestamps using attention weights. The idea: each generated audio token attends to input text tokens, so you can compute a center-of-mass over the attention distribution to estimate when each word is spoken. This works beautifully for encoder-decoder models with cross-attention. It does not work for causal LM self-attention.

Causal self-attention attends to everything that came before. The distributions are diffuse. Every word’s center-of-mass lands somewhere in the middle of the sequence, producing overlapping time ranges that are useless for highlighting.

Sentence-Level Timestamps

The fix was to stop trying to be precise about words and use the precision the model actually gives you.

Narro processes each sentence independently. The number of generated tokens per sentence is known exactly. Multiply by the token duration (2048 samples / 32kHz = 64ms per token) and you get exact sentence timing. No approximation, no heuristics. The timestamps are ground truth.

The alignment JSON is one entry per sentence:

[
  {"text": "Hello world.", "start": 0.0, "end": 0.64},
  {"text": "Goodbye moon.", "start": 0.64, "end": 1.6}
]

I initially tried distributing each sentence’s duration across its words proportionally by character count. It worked, but the DOM manipulation was fragile. Wrapping every word in a <span>, walking the tree to skip headings and code blocks, matching word indices sequentially to alignment entries. If any word got missed or doubled, every subsequent highlight drifted.

Sentence-level is better. The JS player matches sentences to existing <p> elements by substring containment. The DOM stays untouched. The highlighting is a CSS class toggled on the paragraph. Simpler code, no drift, and the visual result is actually clearer to read.

The Browser Side

The JS player runs a binary search on the alignment data at 60fps via requestAnimationFrame and toggles a CSS class on the active paragraph. That is essentially the whole thing. No DOM rewriting, no word wrapping, no skip-tag lists.

What It Sounds Like

Every post on this blog with tts: true in frontmatter has a player at the top. Hit play. The active paragraph highlights as it is spoken. The model runs on CPU in about 20x real-time.

The source is at github.com/queelius/narro.

Universal personal data format. JSONL in, SQL out, MCP to LLMs.

The Format

Every record is a JSON object. All fields optional.

{"mimetype": "text/plain", "content": "I think the key insight is...", "uri": "https://chatgpt.com/c/abc", "timestamp": "2023-05-14T10:30:00Z", "metadata": {"role": "user", "conversation_id": "abc"}}
{"mimetype": "audio/wav", "uri": "file://media/podcast.wav", "timestamp": "2024-01-15", "metadata": {"transcript": "Welcome to...", "duration": 45.2}}
{"mimetype": "image/jpeg", "uri": "file://media/photo.jpg", "metadata": {"caption": "My talk at MIT"}}

The Stack

JSONL files (canonical, portable, human-readable)
    ↓ arkiv import
SQLite database (queryable, efficient, standard SQL)
    ↓ arkiv serve
MCP server (3 tools → any LLM)

Quick Start

pip install arkiv

# Import JSONL to SQLite
arkiv import conversations.jsonl --db archive.db

# Query
arkiv query archive.db "SELECT content FROM records WHERE metadata->>'role' = 'user' LIMIT 5"

# Serve to LLMs via MCP
arkiv serve archive.db

MCP Tools

Why

• Your data lives in silos (ChatGPT, email, bookmarks, photos, voice memos)
• Source toolkits (ctk, mtk, btk, ptk, ebk) export it as JSONL
• arkiv gives you one format, one database, one query interface
• Any LLM can query it via MCP
• JSONL is human-readable and durable. SQLite is the most deployed database in history.

Spec

See SPEC.md for the full technical specification.

GPG signing for static site markdown content.

Sign Hugo/static-site markdown files with GPG, embedding ASCII-armored signatures directly in YAML front matter. Verify authenticity, strip signatures, or check signing status across your content directory.

Install

pip install sigmark

Requires GPG (gpg) to be installed and available on your PATH.

Usage

# Sign all markdown files in a directory
sigmark sign --key you@example.com content/

# Sign a single file
sigmark sign --key you@example.com content/post/hello/index.md

# Verify signatures
sigmark verify content/

# Check signing status
sigmark status content/

# Remove signatures
sigmark strip content/

How It Works

Sigmark signs only the body of each markdown file (everything below the closing --- front-matter delimiter). The GPG signature is stored as a signature field in the YAML front matter:

---
title: Hello World
date: 2026-01-01
signature: |
  -----BEGIN PGP SIGNATURE-----
  iQEzBAABCAAdFiEE...
  -----END PGP SIGNATURE-----
---
Your post body here.

This means front-matter changes (tags, categories, draft status) don’t invalidate the signature, while any change to the actual content does.

Commands

All commands accept files and/or directories. Directories are walked recursively for .md files with YAML front matter. Global flags: --verbose, --dry-run.

License

MIT

Declarative language for constructing complex networks from structured data.

What is netloom?

netloom is a YAML-based DSL that describes how to build a weighted directed graph from structured documents. You declare node types (the units of your graph) and link types (how nodes relate), and netloom constructs a NetworkX DiGraph for analysis.

The output is a directed graph you can analyze with standard tools: community detection, centrality measures, shortest paths, visualization.

import netloom

G = netloom.build("config.yaml")   # returns nx.DiGraph

Or from the command line:

netloom build config.yaml                            # -> output.graphml
netloom build config.yaml -o graph.json --format json
netloom build config.yaml --format gexf

Why not just use a vector DB?

Vector databases are fast nearest-neighbor lookup engines. They answer “what’s similar to X?” netloom answers a different question: “what is the structure of similarity across my corpus?”

netloom wins when:

• Metadata fields (tags, authors, categories) should contribute to similarity, not just filter results
• You care about graph structure: which documents are hubs, which bridge two communities
• You want declarative control over how similarity is composed from multiple signals
• You have multiple node types in the same graph (heterogeneous networks)
• You need directed relationships like citations or containment alongside similarity

Sweet spot: Small-to-medium corpora (<10K documents) where you want to understand structure, not just retrieve.

Source formats

netloom ingests structured data from multiple formats. A single config can combine multiple sources:

source:
  - path: data/conversations.jsonl
    format: jsonl
  - path: data/papers/
    format: json

Markdown is treated as structured data: headings become title, ## sections become a sections list, and the content becomes body. Every record gets a _meta block with full provenance (source path, timestamps, content hash).

Core abstractions

Defaults reduce repetition across node types:

defaults:
  embed:
    model: tfidf

Nodes define the units of your graph. A single source document can produce multiple node types:

nodes:
  conversation:
    from: .
    fields:
      title: { pluck: title }
      tags: { pluck: tags }
    embed:
      field: title              # inherits model: tfidf from defaults

  user_turn:
    from: turns
    where: { role: user }
    fields:
      text: { pluck: text }
    embed:
      field: text

Links define relationships between nodes – similarity, attribute overlap, structural containment, and foreign-key references:

links:
  intent_similarity:
    between: [user_turn, user_turn]
    method: cosine
    min: 0.3

  tag_overlap:
    between: [conversation, conversation]
    method: jaccard
    field: tags

  contains_turns:
    between: [conversation, user_turn]
    method: parent

  cites:
    between: [paper, paper]
    method: reference
    field: references
    target_field: paper_id

Network controls graph construction:

network:
  min: 0.3
  communities:
    algorithm: louvain

Link methods

Symmetric methods produce bidirectional edges. Parent and reference produce directed edges.

Full example

Given a corpus of conversation JSON documents like this:

{
  "id": "conv-2024-0142",
  "title": "Debug authentication middleware",
  "created_at": "2024-11-15T09:23:00Z",
  "model": "claude-sonnet-4-20250514",
  "turns": [
    {
      "role": "user",
      "text": "The auth middleware is rejecting valid tokens after the Redis upgrade",
      "timestamp": "2024-11-15T09:23:00Z"
    },
    {
      "role": "assistant",
      "text": "Let me check the Redis connection config and token validation logic.",
      "timestamp": "2024-11-15T09:23:05Z",
      "tool_calls": ["read_file", "grep"]
    },
    {
      "role": "assistant",
      "text": "Found it -- the Redis key prefix changed from 'session:' to 'sess:' in v7.",
      "timestamp": "2024-11-15T09:23:15Z"
    },
    {
      "role": "user",
      "text": "Ah that makes sense, we upgraded Redis last week. Can you fix it?",
      "timestamp": "2024-11-15T09:23:30Z"
    }
  ],
  "tags": ["debugging", "auth", "redis"],
  "project": "backend-api",
  "outcome": "resolved",
  "tools_used": ["read_file", "grep", "edit"]
}

This netloom config builds a heterogeneous graph where conversations and individual turns are separate node types, connected by semantic similarity, tag overlap, and structural containment:

defaults:
  embed:
    model: tfidf

source:
  path: data/conversations/
  format: jsonl

nodes:
  conversation:
    from: .
    fields:
      id: { pluck: id }
      title: { pluck: title }
      tags: { pluck: tags }
      project: { pluck: project }
    embed:
      field: title

  user_turn:
    from: turns
    where: { role: user }
    fields:
      text: { pluck: text }
      timestamp: { pluck: timestamp }
    embed:
      field: text
      chunking:
        method: sentences
        max_tokens: 256
      aggregate: mean

  assistant_turn:
    from: turns
    where: { role: assistant }
    fields:
      text: { pluck: text }
      tools: { pluck: tool_calls, default: [] }
    embed:
      field: text

links:
  user_intent_similarity:
    between: [user_turn, user_turn]
    method: cosine
    min: 0.3

  cross_role_similarity:
    between: [user_turn, assistant_turn]
    method: cosine
    min: 0.4

  tag_overlap:
    between: [conversation, conversation]
    method: jaccard
    field: tags

  same_project:
    between: [conversation, conversation]
    method: exact
    field: project

  contains_user_turns:
    between: [conversation, user_turn]
    method: parent

  contains_assistant_turns:
    between: [conversation, assistant_turn]
    method: parent

network:
  min: 0.3
  communities:
    algorithm: louvain

Each source document produces one conversation node and multiple turn nodes. The resulting graph has:

• Semantic edges between turns with similar content (cosine similarity)
• Attribute edges between conversations sharing tags (Jaccard) or the same project (exact match)
• Structural edges connecting conversations to their constituent turns (parent links)

Plugin architecture

netloom uses a registry pattern for all pluggable components. Built-in providers and user-written providers are structurally identical:

netloom/
  embeddings/tfidf.py        # built-in, no heavy deps
  embeddings/ollama.py        # local Ollama models
  embeddings/openai.py        # OpenAI API
  metrics/jaccard.py          # built-in
  metrics/cosine.py           # built-in
  chunking/sentences.py       # built-in

Install optional providers via extras:

pip install netloom[openai]
pip install netloom[ollama]

Or write your own:

from netloom import register_embedding

@register_embedding("my-model")
class MyEmbedding:
    def embed(self, text: str) -> list[float]:
        ...

Then reference it in the DSL:

embed:
  field: text
  model: my-model

Use cases

• Conversation analysis: Nodes are conversations and turns. Links are semantic similarity, topic overlap, temporal proximity, structural containment.
• Paper citation networks: Nodes are papers. Links are citation (reference), co-authorship (jaccard), topic similarity (cosine).
• Codebase analysis: Nodes are files, functions, modules. Links are imports, call graphs, semantic similarity.
• Multi-modal documents: Nodes are text chunks, images, tables from the same document. Links are co-occurrence and cross-modal similarity.
• E-commerce catalogs: Nodes are products. Links are semantic description similarity, shared categories, price-range proximity (numeric).

Status

Design phase. The DSL specification is in docs/spec.md. No implementation code yet. We’re refining the design before building.

License

MIT

I wanted to run queries against my medical records. Correlate lab trends with treatment changes. Generate structured question lists before oncology visits. Ask “what changed since my last appointment” and get a real answer. That means getting the data out of the portal and into something programmable.

Chartfold loads EHR exports into SQLite and exposes them to Claude via MCP.

The Problem

In the US, patients can export their medical records. HIPAA and the 21st Century Cures Act guarantee this. What you get depends on the system: Epic MyChart gives you CDA XML files, MEDITECH Expanse gives you FHIR JSON mixed with CCDA XML, athenahealth gives you FHIR R4 Bundles. Different formats, same clinical concepts.

If your hospitals use different EHR systems, none of them have the complete picture. Chartfold merges the exports into one database. But even if you’re at a single hospital, the export format is not something you can work with directly. A directory of CDA XML files is not a database. You can’t query it, chart it, or hand it to an LLM.

The point of Chartfold is to turn whatever your hospital gives you into a SQLite database, then make that database useful.

What It Does

Chartfold is a Python CLI. You point it at an EHR export directory, it parses the XML/FHIR, normalizes everything into a common data model (16 tables, ISO dates, deduplicated), and loads it into SQLite. Then you can query it directly, export it as a self-contained HTML dashboard, or connect Claude to it via MCP.

# Load data from your hospital exports
chartfold load epic ~/exports/epic/
chartfold load meditech ~/exports/meditech/

# Query directly
chartfold query "SELECT test_name, value, result_date FROM lab_results
                 WHERE test_name LIKE '%CEA%' ORDER BY result_date DESC"

# Export a self-contained HTML file
chartfold export html --output chartfold.html

# Start the MCP server for Claude
chartfold serve-mcp

The Claude Integration

This is why Chartfold exists for me.

The MCP server exposes the database to Claude Code. Setup is one file. Drop a .mcp.json in any directory where you run Claude Code:

{
  "mcpServers": {
    "chartfold": {
      "command": "python",
      "args": ["-m", "chartfold", "serve-mcp", "--db", "/path/to/chartfold.db"]
    }
  }
}

That’s it. Claude now has read access to your entire medical history via SQL, plus tools for saving notes and structured analyses. I keep my database in a private directory and my .mcp.json pointing at it. Open Claude Code, and I’m talking to my records.

The kinds of things I actually use it for:

“What’s changed since my last oncology visit on January 15?”

Claude writes SQL, reads the results, and gives me a structured diff: new lab results, new imaging, changed medications, new clinical notes.

“Generate a prioritized question list for my appointment with Dr. Tan tomorrow.”

Claude reads my recent labs, imaging reports, pathology, and genomic results, then produces a tiered document organized by clinical urgency.

“Show me my CEA trend and flag any inflection points.”

Claude queries the lab_results table, filters by test name, and walks through the time series.

The analyses get saved back to the database (via dedicated MCP tools) and appear in the HTML export as tagged, searchable documents.

Here’s what one looks like expanded: a structured question list for an oncology appointment, organized by urgency tier, referencing specific test results and treatment options.

I use this before every oncology visit. When you have 1776 lab results, 53 imaging reports, and 9 pathology reports, you need something to synthesize them. That’s what Claude does well, but it needs structured data to work with. Chartfold provides the structured data. Claude provides the reasoning.

The MCP server exposes 25 tools. Here’s the full list:

Clinical data is read-only (the SQLite connection opens in ?mode=ro, enforced at the engine level). Claude can’t modify your clinical records, only read them and save its own notes and analyses.

Most of these tools exist so Claude doesn’t have to write SQL for common tasks. But run_sql is the escape hatch: anything the specialized tools don’t cover, Claude can query directly.

The HTML Dashboard

The HTML export embeds the entire SQLite database using sql.js (SQLite compiled to WebAssembly). Open the file in a browser and you get an interactive dashboard. Everything runs client-side. No server, no cloud, no account. The file never phones home.

Lab Charts

Lab charts show time-series data across sources with reference ranges. Here, creatinine and albumin are tracked over four years across MEDITECH (blue) and Epic (orange).

For patients who do deal with fragmented records across incompatible systems, this is the view that doesn’t exist in any single portal. The charts are configurable via a TOML file, which you can auto-generate from your data:

chartfold init-config

Conditions

Conditions with ICD-10 codes, onset dates, and source provenance.

Medications

Medications show “Multi-source” badges when the same drug appears in multiple systems.

Imaging

Imaging reports with the full narrative findings. Useful for visit prep: search for a specific study, read the impression, bring context.

Source Documents

Source PDFs and scanned documents grouped by date.

SQL Console

For anything the UI doesn’t cover, there’s a SQL console. Every table, every column, every index.

Dark Mode

Architecture

Three-stage pipeline, each stage independently testable:

Raw EHR files (CDA XML, FHIR JSON, CCDA XML)
    |
    v
[Source Parser]  -- format-specific extraction
    |
    v
[Adapter]        -- normalize to UnifiedRecords (16 dataclass types)
    |
    v
[DB Loader]      -- idempotent upsert into SQLite

Source parsers handle the XML/FHIR parsing. Adapters normalize dates to ISO 8601, parse numeric values, deduplicate, and map into a common data model. The DB loader uses upsert with natural keys, so re-running a load is safe.

After loading, the CLI prints a stage comparison table: parser count, adapter count, DB count. If the numbers don’t match, you know where data was lost.

Currently supports Epic (CDA), MEDITECH (FHIR + CCDA), and athenahealth (FHIR R4). These importers were written against my own exports. I can’t guarantee they’ll work for yours. EHR exports vary by site, software version, and configuration. The pipeline is designed as a plugin system for exactly this reason: adding a new source means writing a parser, an adapter, and wiring them into the CLI. The CLAUDE.md has a recipe, and Claude can write a new importer from a sample export in about an hour.

Export Formats

• HTML SPA: self-contained single file with embedded SQLite, Chart.js, and sql.js. No external dependencies. Copy it to a USB drive.
• Markdown: visit-focused summary with configurable lookback, optional PDF via pandoc.
• JSON: full-fidelity round-trip format. Export, then import to a new database with identical record counts.
• Hugo site: static site with detail pages and cross-linked records.
• Arkiv: universal record format (JSONL + manifest) for long-term archival.

The HTML export is a single file. No server, no backend, no account. You can host it on a static site (I host mine on GitHub Pages), email it to a family member, or hand it to a doctor on a USB drive. Because it’s just a file, you can protect it with PageVault to add password-based encryption before sharing. The recipient opens the file, enters the password, and gets the full interactive dashboard. No server involved at any step.

Medical records should not depend on someone else’s infrastructure. A single HTML file with an embedded database and WebAssembly runtime is about as durable as digital data gets.

Getting Started

pip install chartfold

# Load your exports
chartfold load auto ~/path/to/export/

# Query
chartfold query "SELECT test_name, value, result_date FROM lab_results LIMIT 10"

# Export
chartfold export html --output my-records.html

# Connect Claude
chartfold serve-mcp

The code is on GitHub: queelius/chartfold. Python 3.11+, depends on lxml and not much else.

Chartfold started because I wanted to ask questions about my own medical records and couldn’t. Now I can.

I read a post arguing that the most valuable programming skill in 2026 is deleting code. The thesis: AI generates code faster than anyone can review it, so the real value is in curation and subtraction. Code is a liability, not an asset.

I agree with the observation. I disagree with the prescription.

The Thesis

The argument is straightforward. AI tools can produce entire modules in the time it takes to write a spec. Codebases are accumulating features nobody asked for, abstractions nobody needs, and boilerplate that exists because the model defaulted to verbosity. Teams that used to struggle to ship enough code now struggle with too much of it. In this world, the programmer’s role shifts from writer to editor. The most valuable activity becomes knowing what to cut.

I’ve seen this. Projects accumulate code the way attics accumulate boxes. Nobody remembers why half of it is there. It sits untouched for months, adding to cognitive load, making every change harder to reason about. When you finally clear it out, nothing breaks and the rest becomes legible again. Code rot is real. AI accelerates it. The instinct to subtract is correct.

But subtraction is symptom treatment. The underlying problem isn’t volume. It’s that most code doesn’t know why it exists. It was written (or generated) to solve a local problem, it solved it or half-solved it, and then it sat there, disconnected from any larger purpose. Code without purpose is what bloats. Deletion is the right instinct pointed at the wrong layer. The cause isn’t too much code. It’s too little intent.

What I Found Instead

I started building tools two years ago because I needed my personal data to survive even if I wasn’t around to maintain it. That’s the whole constraint. Whatever I build has to work without me.

The first tool was a conversation archiver. I had years of AI conversations across ChatGPT, Claude, and Copilot trapped in platforms that might not exist next decade. I needed them in formats that degrade gracefully. SQLite for structured queries. JSONL for streaming and interchange. Markdown for human reading. If the tool disappears, the data is still a file you can open with anything. If SQLite disappears, the JSONL is still searchable with grep. If everything disappears, the Markdown is still readable in a text editor. Each layer works without the one above it.

Then I needed the same thing for bookmarks. Then ebooks. Then photos, email, medical records, notes. Each tool exists because the previous one exposed a gap. The medical records needed secure sharing without a server. The whole collection needed a dead man’s switch. The archive eventually needed something stranger: a way to answer questions after I couldn’t.

None of this was planned as an ecosystem. I built the next thing I needed, and the next, and the next. At some point I looked back and realized they all pointed the same direction. This is a life project. Everything serves one purpose.

The Stack

One purpose: durable personal data that outlasts its creator.

Philosophy

• longecho: The Long Echo specification. Self-describing data, durable formats, graceful degradation.

Universal Format

• arkiv: Universal personal data format. JSONL in, SQL out, SQL back to JSONL. Its MCP server can host any collection intelligently, regardless of domain. One format, one database, one query interface.

Source Toolkits

• memex / ctk: AI conversations. Import, query, continue in the browser, export durable archives.
• A family of domain toolkits for bookmarks, ebooks, photos, email, and notes. Different domains, identical architecture.
• chartfold: Medical records from three hospital systems, consolidated into one queryable database.

Infrastructure

• pagevault: Client-side encryption for any HTML file. No server.
• posthumous: Federated dead man’s switch.
• repoindex: Index and query across ~120 git repos.
• A collection of Claude Code plugins: MCP servers, agents, and skills that wire everything into my daily workflow.

The Endgame

• eidola: A conversable persona assembled from all of the above. Its first form is a Claude Code plugin backed by the combined archive. Not resurrection. An echo.

The Pattern

Every tool follows the same architecture. SQLite for storage. CLI for local use. MCP server for Claude, or the CLI wrapped in a light Claude Code plugin. Export to self-contained HTML you can host anywhere or open from a file. Export to longecho-compliant archives that work without the tool. The data always outlasts the software.

Take memex. Import your AI conversations, query them with SQL, talk to Claude about them via MCP, or export a single HTML file where you can browse and continue conversations in the browser. Download the SQLite from that same page, and you’re back to durable local data. The cycle closes. This is how most of the tools work.

Arkiv sits in the middle. The source toolkits produce JSONL. Arkiv imports it to SQLite. Arkiv exports it back to JSONL. Its MCP server can expose any collection to Claude, regardless of what domain it came from. The data flows in a circle, always in formats that describe themselves.

Source toolkits → arkiv → longecho
                              ↓
                  pagevault (encrypt) + posthumous (deliver)
                              ↓
                         eidola (echo)

I have never once needed to delete one of these tools. Not because I’m a better programmer than anyone else. Because each one exists for a specific reason that connects to the others. When code has purpose, dead weight doesn’t accumulate.

This isn’t architectural foresight. It’s what happens when you build from a clear constraint. “My data has to survive without me” is a filter that works at the design stage. Every tool either serves that constraint or it doesn’t get built. There is no third category.

The Actual Skill

The most valuable skill isn’t deleting code or writing it. It’s knowing why you’re building.

If you know the purpose, code stays minimal because unnecessary code doesn’t serve it. You don’t need periodic purges. The purpose does the culling before anything gets written. Deletion is what happens when purpose was absent from the start. It’s retrospective correction for a problem that clear intent would have prevented.

I know what I’m building toward. The tools will echo after I stop maintaining them. That was the point.

A paradigm for computing with values whose meaning is hidden behind a one-way trapdoor.

The Idea

A cipher map is a total function on bit strings. Every input produces output. The concepts of “domain,” “correct,” and “incorrect” only exist on the trusted machine that holds the decoder. The untrusted machine sees only opaque bits flowing through opaque lookup tables — it cannot distinguish signal from noise.

Foundations

The foundations/ directory contains the authentic source documents (2023-2024) that define the framework:

• bernoulli-model.md — The Bernoulli Model: latent/observed duality, confusion matrices, HashSet construction achieving -log_2(epsilon) bits per element
• noisy-gates.md — Noisy Turing Machines: composing Bernoulli logic gates, interval arithmetic for error propagation through circuits
• trapdoor-boolean-algebra.md — Boolean Algebra Over Trapdoors: approximate homomorphism from powerset to bit strings via cryptographic hash, marginal uniformity
• entropy-maps.md — Entropy Maps: prefix-free hash codes for function approximation, mu = H(Y) space bound, two-level hash construction

Four Properties of a Cipher Map

A cipher map f-hat implementing a latent function f satisfies:

1. Totality — f-hat is defined on all inputs (bits in, bits out). Out-of-domain inputs produce random output. With probability epsilon, random output happens to be a valid codeword.
2. Representation Uniformity — Each domain value has multiple encodings. The distribution over encodings is delta-close to uniform, preventing frequency analysis.
3. Correctness — For in-domain inputs, decode(f-hat(encode(x))) = f(x) with probability at least 1 - eta.
4. Composability — Composing cipher maps compounds error predictably: eta_total = 1 - (1 - eta_f)(1 - eta_g).

Author

Alexander Towell — metafunctor.com

A forge for Claude Code plugins.

Claude Anvil is a plugin marketplace containing nine Claude Code plugins for academic research, creative worldbuilding, publication workflows, site management, repository intelligence, personal metadata, encryption, and more. Each plugin provides skills, commands, and agents that extend Claude Code with domain-specific capabilities.

Quick Start

# Install the marketplace
/plugin marketplace add queelius/claude-anvil

# Install an individual plugin
/plugin install papermill@queelius

Plugins

Plugin Anatomy

Each plugin follows Claude Code plugin conventions:

<plugin>/
├── .claude-plugin/plugin.json   # Manifest: name, version, description, author
├── skills/<name>/SKILL.md       # Interactive skills (the core logic)
├── commands/<name>.md           # Slash commands (thin wrappers or rich docs)
├── agents/<name>.md             # Autonomous subagents with system prompts
└── hooks/hooks.json             # Event handlers (optional)

Skills are the heart of each plugin — they contain the domain knowledge and workflow logic. Commands are thin wrappers that trigger skills via /plugin:command syntax. Agents run autonomously for tasks like literature surveys or code review.

Not every plugin uses every component. Minimal plugins like deets have only a skill and a command. Larger plugins like worldsmith add hooks for ambient project detection and propagation discipline.

License

MIT

The problem: a series system fails when any component fails. You observe system-level failure times. But you often cannot tell which component caused the failure (masking). Some systems are still running when testing ends (censoring). Given this incomplete data, estimate component reliability.

This is not a tutorial. It is a map of where things stand and where they are going.

How It Started

I picked this for my math master’s thesis because it combined everything I wanted to learn: survival analysis, mixture models, the EM algorithm, bootstrap inference, simulation studies. I focused on Weibull components with right censoring, but the likelihood derivation was distribution-agnostic. The general framework was there; I just did not treat it as the main contribution.

The implementation was one monolithic R package (wei.series.md.c1.c2.c3). It worked. It was not reusable. The distribution algebra, the MLE infrastructure, the likelihood model, the series system logic, the masking conditions: everything tangled together. Want to change the distribution family? Rewrite half the package.

I defended in October 2023. The thesis was fine. The code was a dead end.

Pulling It Apart

After the defense, I started asking: what are the actual algebraic structures here?

It took a while, but the answer was several independent layers:

1. Distributions form an algebra. Add, scale, compose. That became algebraic.dist.

2. MLEs form an algebra. Delta method, bootstrap, reparameterization. That became algebraic.mle.

3. Likelihood models are composable. Heterogeneous data is a sum of likelihood contributions, one per observation type. That became likelihood.model.

4. MLE solvers are composable. Chain them, race them, add restarts. The SICP closure property: composing solvers gives you a solver. That became compositional.mle.

5. Hazard functions define distributions. Write \(h(t; \theta)\), get \(S(t)\), \(F(t)\), \(f(t)\), quantiles, sampling, and MLE for free. That became flexhaz. A companion, serieshaz, composes component hazards into series system distributions.

The dependency graph:

                algebraic.dist
                 |          \
             algebraic.mle    \
              |        \       \
compositional.mle   likelihood.model
                      |         \
                   flexhaz    maskedcauses
                     |
                  serieshaz
                     |
                  maskedhaz

None of this was planned. It emerged from repeatedly asking “why is this code so hard to change?” until the natural joints appeared.

AD and nabla

Preparing the master’s defense (which included BCa bootstrap confidence intervals) exposed the places where my understanding was mechanical rather than structural. I knew the formulas. I did not always know why they were the right formulas.

After the defense, I experimented with PyTorch’s autograd for MLE. I was already familiar with the relationship between gradients and score functions, but running it in a computational graph solidified it. The score function is the gradient of the log-likelihood. The observed information is the negative Hessian. Same computation, different names.

That led me to build nabla, which I think is the most general AD package in R. Nested dual numbers, S4 dispatch. D(f) gives gradients. D(D(f)) gives Hessians. Works through loops and branches. Arbitrary order, machine precision.

It is too slow for optimization. That is not the point. You find your MLE however you want (analytical, optim, whatever), then use nabla to characterize the solution: observed Fisher information, skewness analysis, higher-order diagnostics. The right tool for final analysis, not for search.

The Foundation Paper

In 2025, I wrote the paper that should have been separate from the start.

The thesis had the distribution-agnostic likelihood derivation buried in a document mostly about Weibull. The foundation paper extracts that general framework and gives it a proper treatment. It handles all four observation types (exact, right-censored, left-censored, interval-censored) and defines three conditions on the masking mechanism:

• C1: The candidate set always contains the true failed component.
• C2: Masking probabilities are symmetric across components.
• C3: Masking probabilities do not depend on the system parameters.

Under C1-C2-C3, the likelihood factors cleanly. The paper derives the general likelihood, score equations, and Fisher information, then instantiates them for exponential, Weibull, Pareto, log-normal, and gamma families.

The exponential companion pushes the simplest case as far as the math goes. Closed-form MLE. Analytical Fisher information. A proof that information loss from masking is monotone (more masking, less information, strictly). And a result I did not expect: uniform masking maximizes the entropy of the candidate set given the failure cause. It is the worst case for identifiability among all C2-compliant masking models. The closed-form results are not just a convenient special case. They are pessimistic bounds.

Where Things Are

Software

Six packages on CRAN, five more in progress:

maskedcauses and maskedhaz solve the same problem at different levels of generality. maskedcauses has closed-form solutions for exponential and Weibull. maskedhaz handles arbitrary hazard functions via numerical integration. When both are installed, the test suites cross-validate.

mdrelax explores relaxing the C1, C2, and C3 conditions: informative masking, parameter-dependent masking, masking probabilities less than one. Right now it is research code for a paper. If the results hold up, a proper package may come out of it.

Papers

What I Am Working On Next

I have five companion paper directions. Some are real; some are barely more than a paragraph.

Identifiability (active)

The foundation paper proves a necessary and sufficient condition for identifiability from masked data. But it is a yes/no result. It says nothing about how much diagnostic separation you need in practice, which candidate set structures help most, or how identifiability degrades as masking gets heavier.

I want to give a graph-theoretic characterization (confounding graph, connected components as “super-components”), a linear algebra condition for the exponential case (rank of the candidate-set matrix), and simulation ablation studies.

The exponential companion connects directly: uniform masking at \(w = m-1\) is the most pessimistic identifiable scenario. So the question inverts: what masking design minimizes information loss? That is a D-optimality problem for the masking mechanism. I do not know the answer yet.

Observation scheme composition (idea)

maskedcauses implements composable observation functors: observe_right_censor(), observe_left_censor(), observe_periodic(), observe_mixture(). The mathematical content is that C1-C2-C3 are preserved under these compositions. If your masking mechanism satisfies the conditions, and you compose it with an independent censoring scheme, the composition still satisfies them.

The proofs are straightforward. The interesting question is whether there is a category-theoretic formulation. Observation schemes as morphisms, composition as functor composition. If so, the closure theorems become instances of a general principle rather than case-by-case arguments. I do not know if this leads anywhere useful or is just abstraction for its own sake.

Parsimony vs physical structure (idea)

If the data are heavily masked and the sample is small, a single Weibull fits the system lifetime data about as well as an \(m\)-component series model. Standard model selection says: use the simpler model. Engineering knowledge says: the components exist.

This is a real tension. If you want to predict system lifetimes, maybe the single Weibull is fine. If you want to estimate component reliability for maintenance planning, the series decomposition matters even if it is over-parameterized relative to the data. I think this is a short paper. Maybe just an essay with simulations.

Weibull companion (planned)

The exponential companion has closed-form results because exponential lifetimes are memoryless. Weibull has shape parameters, which means time-varying hazards and numerical integration for some censoring types.

The thesis covers Weibull estimation. The companion paper would redo it properly in the foundation paper’s notation: score equations, Fisher information, simulation studies, guidance for choosing homogeneous vs heterogeneous models. The theory exists in pieces across the thesis and maskedcauses. Writing it up as a standalone paper is substantial work.

CRAN and JOSS submissions

The immediate practical work. flexhaz, serieshaz, maskedcauses, and maskedhaz are all targeting both CRAN and JOSS. I have been through the CRAN gauntlet six times. It is tedious but not mysterious.

What I Have Learned

Three things.

Decompose first. The monolithic thesis code worked but could not grow. Pulling it into algebraic layers took months. Every piece became independently testable, reusable, publishable. The decomposition is itself a research contribution, not just engineering convenience.

Publish the general theory separately. The thesis had the distribution-agnostic framework buried in a Weibull-specific document. Writing the foundation paper forced me to separate what is structural (C1-C2-C3 factorization, observation type taxonomy) from what is distribution-specific. That separation made the companion papers possible.

The Fisher information matrix is the right lens. For two years I treated FIM as “the thing you invert to get standard errors.” The exponential companion forced me to see it as a measure of how much information the observation scheme actually provides. Once I saw that, the identifiability results, the information loss monotonicity, and the optimal design questions all fell out naturally. FIM connects the statistical theory to the practical question of how to design better diagnostics.

The work is not close to done. But the pieces are in place: a clean theoretical framework, a modular software stack, two completed papers, a clear map of what comes next. Most days, that is enough to keep going.

A Claude Code plugin for academic research paper lifecycle management.

Papermill provides interactive skills and autonomous agents that cover the full pipeline from idea to submission: thesis refinement, literature surveys, experiment design, editorial review, and venue matching.

Installation

claude plugins add ~/github/papermill

Or from any directory containing the plugin:

claude plugins add .

Commands

Invoke commands with /papermill:<name> in Claude Code. Each command triggers its corresponding skill.

Agents

Agents run autonomously and produce structured output files.

State File

Papermill uses a .papermill.md file in each paper repository to persist state across sessions. This file has YAML frontmatter (structured metadata) and a markdown body (session notes).

The state file is created by /papermill:init and updated by other skills as you work.

Workflow

A typical workflow:

1. /papermill:init – Set up the state file in your paper repo
2. /papermill:thesis – Crystallize your central claim
3. /papermill:prior-art – Survey the literature, identify gaps
4. /papermill:outline – Design the paper structure
5. Write the paper (papermill helps, but the writing is yours)
6. /papermill:experiment / /papermill:simulation – Design computational work
7. /papermill:proof – Develop and verify proofs
8. /papermill:review – Get editorial feedback
9. /papermill:venue – Choose where to submit
10. /papermill:polish – Final pre-submission check

Skills can be used in any order and revisited as needed. Use /papermill:status at any time for orientation.

License

MIT