Study notes: The Demon Is a Learner: The Mathematics

With time measured in the slow unit \gamma/k, the equation becomes \varepsilon X'' + X' = \varphi(X) with \varepsilon = mk/\gamma^2. As a system it is X' = U, \varepsilon U' = \varphi(X) - U. For fixed X the fast equation has the equilibrium U = \varphi(X) with eigenvalue -1/\varepsilon < 0, so it attracts. After an initial layer of duration O(\varepsilon), solutions follow X' = \varphi(X), that is \dot x = F/\gamma. For the spring the roots are \lambda_\pm = (-1\pm\sqrt{1-4\varepsilon})/2\varepsilon. At \varepsilon = 0.1 they are -1.127 and -8.873, against the reduced rate -1. At the threshold \varepsilon = 1/4 both equal -2, so the first-order law is off by a factor of 2 there.

W = \binom{20}{10} = 184{,}756 and \binom{20}{5} = 15{,}504, so \ln W is 12.13 and 9.65. N\,H(p) is 20\ln 2 = 13.86 and 20\times0.562 = 11.25. Stirling’s formula is rough at N = 20, and the differences agree better: 2.48 against 2.62. Boltzmann’s entropy \ln W is Shannon’s entropy per ball times N. It is the length of the description still lacking once the macrostate is known. A macrostate with H lower by a fixed \delta has fewer microstates by a factor of e^{-N\delta}, and with N near 10^{23} no fluctuation reaches it.

The counts of microstates multiply and the entropies add, so the likeliest split maximizes S_A(E_A) + S_B(E_{\text{tot}} - E_A), which gives dS_A/dE_A = dS_B/dE_B. That shared quantity defines 1/T = \partial S/\partial E. T is the energy that buys one nat, and heat Q raises a bath’s entropy by Q/T. A system that changes by \Delta E and \Delta S with no work done hands the bath heat -\Delta E, so \Delta S_{\text{tot}} = \Delta S - \Delta E/T = -\Delta F/T. The second law for system and bath together says that F does not rise. For the two states, Z = 1 + e^{-1} = 1.368 and p = (0.731,\ 0.269), so E = 0.269, S = 0.582 and E - TS = -0.313 = -\ln Z.

For any function \varphi of velocity, \frac{d}{dt}\int\varphi f\,dv is an integral of \varphi\,[f'f_*' - ff_*]\,B over collisions. Two relabelings keep its value, each with unit Jacobian and each leaving B unchanged. Swapping the balls replaces \varphi by \varphi_*. Exchanging a collision with its reverse replaces it by \varphi' and flips the sign of the bracket. Averaging the four forms replaces \varphi by \frac14(\varphi + \varphi_* - \varphi' - \varphi_*'). With \varphi = \ln f + 1 the integrand is -\frac14(f'f_*' - ff_*)\ln\frac{f'f_*'}{ff_*}\,B \le 0, because (a-b)\ln(a/b)\ge 0. Equality needs ff_* = f'f_*' for every collision, so \ln f adds up to the same total before and after, a combination of mass, momentum and energy: the Maxwell distribution f\propto e^{-|v-u|^2/2T}.

The swap and the reverse collision hold in either direction of time. The ansatz does not. Assume independence on outgoing pairs instead. Each incoming pair continues as the outgoing pair with the same density, so the loss term becomes f'f_*' and the gain term ff_*, the right side of Boltzmann’s equation changes sign, and H rises. That theorem is equally rigorous, and both assumptions can hold at once only in equilibrium. Mechanics does not choose between them, which is what Loschmidt’s reversal, Zermelo’s recurrence and Burbury’s Condition A each point out. So the arrow lies in the initial conditions: a gas prepared by ordinary means has no correlations tuned to its future. Extended backward from an uncorrelated state, entropy would rise into the past too. The direction comes from placing the special state at the beginning.

Define the total correlation C(t) = N\,S(t) - S_N(t) \ge 0, the divergence of the joint distribution from the product of its marginals. S_N is constant by Liouville’s theorem and S_N(0) = N\,S(0), so N\bigl(S(t) - S(0)\bigr) = C(t). Every nat gained, summed over the balls, is a nat stored in correlations among them. Nothing has been lost from the gas. The ansatz is then an act of forgetting: it says that each incoming pair carries none of the correlation that earlier collisions made. Entropy rises because a bounded description keeps f and discards the rest.

For any two distributions and any kernel, D(\mu P\,\|\,\pi P) \le D(\mu\,\|\,\pi). The proof is the chain rule: the divergence between the joint laws of a state and its successor equals D(\mu\|\pi), because both use the same kernel, and it also equals D(\mu P\|\pi P) plus a nonnegative term for the backward conditionals. Here the stationary distribution is \pi = (2/3,\ 1/3). D(\mu\|\pi) = \ln 1.5 = 0.405, and after one step \mu P = (0.9,\ 0.1) with divergence 0.150. The arrow enters with a kernel chosen without reference to \mu_t: the randomness applied at each step knows nothing of the distribution it acts on, which is the Stosszahlansatz for a bath. For stochastic gradient descent the assumption is that each minibatch is drawn fresh. Entropy itself is guaranteed to rise only when \pi is uniform.

The exact equation has the force -V', a friction with memory -\int_0^t\Gamma(t-s)\,\dot x(s)\,ds, and a term \xi(t) that is a fixed function of the bath’s initial state. \xi becomes noise when we stop tracking those initial conditions. With the bath started in its Gibbs state, \langle\xi(t)\,\xi(s)\rangle = T\,\Gamma(t-s): noise and friction have the same kernel because they come from the same oscillators. This is the fluctuation–dissipation relation, derived and not assumed. In the limit of a continuous bath \Gamma\to2\gamma\,\delta, and the equation is Langevin’s. The arrow came in with a bath that started uncorrelated with the particle and has too many frequencies for anything to come back. With evenly spaced frequencies a distance \Delta apart, the cosines realign at t = 2\pi/\Delta and a particle that had come to rest is kicked back up, which is Zermelo’s objection in a bath.

Balance. With probability current J = -\gamma^{-1}(\rho\nabla L + T\nabla\rho), the system’s entropy rate plus the heat rate over T is \int\gamma|J|^2/(T\rho) \ge 0. Integrated over a process, \Delta S_{\text{sys}} + Q/T \ge 0: if the system’s entropy falls by \Delta nats, at least T\Delta of heat goes to the bath, and one bit is \ln 2 nats.

Free energy. Substituting \ln\pi = -\beta E - \ln Z gives F[\rho] = F_{\text{eq}} + T\,D(\rho\|\pi). An unknown bit is spread evenly over two wells, in equilibrium. Resetting it reaches a state concentrated in one well, for which D(\rho\|\pi) is higher by \ln 2, so it takes work of at least T\ln 2. The mean energy is unchanged, so the work leaves as heat. Run the other way, this is Szilard’s engine.

In joules, 1.38\times10^{-23}\times300\times\ln 2 = 2.9\times10^{-21}.

Writing dissipates at least T\,I_{\text{mem}} and extraction turns at most T'\,I_{\text{rel}} of heat into work, with I_{\text{mem}} = I[M, X] and I_{\text{rel}} = I[M, Y]. The best net work is T'\,I_{\text{rel}} - T\,I_{\text{mem}} on a heat intake of T'\,I_{\text{rel}}, so \eta = 1 - \frac{T}{T'}\frac{I_{\text{mem}}}{I_{\text{rel}}}: Carnot’s limit minus \frac{T}{T'}\frac{I_{\text{irrel}}}{I_{\text{rel}}}. The engine produces work only if I_{\text{rel}}/I_{\text{mem}} \ge T/T'. The net heat dissipated over a cycle is at least T\,I_{\text{mem}} - T'\,I_{\text{rel}}. Minimizing that bound over p(m\mid x) is \min\ I[M, X] - \alpha\,I[M, Y] with \alpha = T'/T, the information bottleneck. Splitting I_{\text{mem}} = I_{\text{irrel}} + I_{\text{rel}} shows the objective as I_{\text{irrel}} - (\alpha - 1)\,I_{\text{rel}}: an irrelevant nat costs 1, and a relevant nat earns \alpha - 1 net of its own writing.

Lower bound. The shortest programs form a prefix-free code, so by Kraft’s inequality their average length is at least H(P).

Upper bound. Describe x by a description of P followed by the Shannon–Fano codeword for x under P, of length \lceil-\log_2 P(x)\rceil. Averaging gives H(P) + K(P) + O(1).

Dynamics. A program for x_t can consist of a shortest program for x_0, self-delimiting descriptions of f and of t, and a fixed routine that applies f t times. So K(x_t) \le K(x_0) + K(f) + K(t) + O(1), with K(t) \le \log_2 t + 2\log_2\log_2 t + O(1). If f is a computable bijection the argument runs backward too, so the complexity can neither rise nor fall by more than these terms. At the moment of reversal the gas of Figure 2 has the complexity of its initial state, up to these terms, although its one-ball statistics have nearly reached the Maxwell value.

• Partition function. Its logarithm gives means and covariances by differentiation: softmax, exponential families, Fisher information, the two-phase learning rule.
• Gibbs’s variational principle. The evidence lower bound is the negative of a free energy; also EM, regularized objectives and PAC-Bayes.
• Legendre transform. Conjugate duality; a penalty and a trust region are Legendre partners; a cost-function market is path-independent.
• Laplace’s method. The Occam factor, BIC, flat minima.
• Fluctuation and response. Score-function gradients, and a temperature for SGD, T_{\text{eff}} = \eta\sigma^2/2B.
• Detailed balance. MCMC, simulated annealing and SGLD; only ratios of \pi appear, so Z cancels.
• Jarzynski and Crooks. Annealed importance sampling, with the gap of the evidence lower bound as dissipated work.
• Thermodynamic length. Schedules for annealing and for the noise of diffusion models.
• Mean field and the cavity. Variational inference, belief propagation, thresholds of capacity.
• Time reversal. Diffusion models.

The paths read backward are a diffusion with the same noise and the drift \nabla V + 2T\,\nabla\ln\rho_t (Anderson). To check, the forward Fokker–Planck equation in reverse time is \partial_s\rho = -\nabla\cdot(\rho\nabla V) - T\Delta\rho, and -T\Delta\rho = -\nabla\cdot(2T\rho\,\nabla\ln\rho) + T\Delta\rho, so the stated drift with the same noise gives the same path of densities. With \pi\propto e^{-V/T} the drift is -\nabla V + 2T\,\nabla\ln(\rho_t/\pi). At equilibrium the second term vanishes and the reversed film is the same process, which is detailed balance. Out of equilibrium the correction depends on \rho_t, the dependence §9.7 denies to a bath’s kernel, and the forward process dissipates exactly as fast as the reverse needs correcting. A trained network holds the score \nabla\ln\rho_t as statistical knowledge of a population and looks at nothing. A demon gains work from what it knows, while a physical run of the reversal would cost at least T\,D(\rho_{\text{data}}\|\pi) of work, the total entropy production of the forward run.

• The T of a regularized objective is statistical. It has no bath and pays no joules.
• A rescaling of a rectifier network’s weights makes a minimum as sharp as you like without changing the function. The Hessian depends on coordinates. Width alone is not the invariant, and width measured against a prior is.
• T_{\text{eff}} is derived for isotropic gradient noise. Anisotropic noise means the stationary state is not Gibbs.
• The formula assumes a full relaxation at each step, which a diffusion model does not have. What carries over to noise schedules is the path and its Fisher length.
• Rare paths with low work dominate the average. An accurate estimate needs a number of samples of the order of e^{D}, with D the dissipation of the reverse process. For the evidence lower bound that is D(p(z\mid x)\|q), the divergence variational inference does not minimize.