Automatic-Differentiation

Duality: The Hidden Structure of Opposites

Many structures come in pairs: forward/reverse AD, push/pull iteration, encode/decode. Recognizing duality lets you transfer theorems and insights between domains.

Differentiation: Three Ways

Three approaches to computing derivatives, forward-mode AD, reverse-mode AD, and finite differences, each with different trade-offs for numerical computing and machine learning.

Reverse-Mode Automatic Differentiation

Reverse-mode automatic differentiation is just the chain rule applied systematically. I built one in C++20 to understand what PyTorch and JAX are actually doing.

Forward-Mode Automatic Differentiation

Dual numbers extend the reals with an infinitesimal epsilon where epsilon^2 = 0. Evaluate f(x + epsilon) and you get f(x) + f'(x)*epsilon. The derivative falls out of the algebra.

Femtograd: Like Micrograd, But Worse