Information-Geometry

Fisher Flow: An Information-Geometric Framework for Sequential Estimation

Fisher Flow: Optimization on the Statistical Manifold

Gradient descent in Euclidean space ignores the geometry of probability distributions. Natural gradient descent uses the Fisher information metric instead. Fisher Flow makes this continuous.

Your Optimizer Is (Approximately) Propagating Fisher Information

Adam, K-FAC, EWC, and natural gradient are all approximating the same thing at different fidelity levels. The math and the caveats.