Deep learning rediscovered renormalization

I met the renormalization group in a stat-mech lecture at TUM, on a Tuesday, hungover, half-convinced I was in the wrong room. The professor drew a square lattice of spins on the board, then drew a coarser lattice on top of it, each new spin standing in for a block of the old ones. He averaged the block down to a single value, threw away everything inside it, and asked what the system looked like now. Then he did it again. And again. The point, he said, was that most of what you wrote down at the fine scale does not survive the zoom-out. The universe keeps a tiny number of knobs and discards the rest, and the trick is knowing which ones it keeps.

That last part lodged itself somewhere and would not leave.

Throwing away exactly the right details

The renormalization group is, at heart, a disciplined way of forgetting. You start with a microscopic description, far more detailed than you can hold in your head, and you build a sequence of coarser descriptions by integrating out the short-distance fluctuations. Block the spins. Rescale. Renormalize the couplings. Repeat. What flows out the other side are the relevant parameters, the handful of quantities that actually control the large-scale behavior, while the irrelevant ones die off under the iteration like noise washing out of an average.

This is the machinery behind one of the most unreasonable facts in physics: universality. Water boiling and a magnet losing its magnetism near their critical points are described by the same exponents, even though one is molecules and the other is electron spins. They share nothing microscopically. They share a fixed point of the coarse-graining flow. The details were never the physics. The flow was.

A network does the same thing, and nobody told it to

Years later I am staring at the intermediate activations of a convolutional network, the kind of thing you visualize when a training run is misbehaving and you have run out of better ideas. Early layers fire on edges and little oriented gradients. The next ones assemble those into textures and corners. Deeper still, something that responds to a wheel, a face, a whole object. Each layer takes the representation below it, pools and mixes it, and hands up something coarser and more abstract. Fine detail gets discarded on the way. What survives is what matters at the next scale.

I had seen that shape before, on a Tuesday, on a blackboard.

The resemblance is not hand-wavy decoration. Pooling is literally a block average. Depth is literally an iterated map, the same transformation applied over and over with the scale changing underneath you. A deep network learns which features are worth carrying upward and which to throw away, which is the entire job description of the renormalization group. When I first noticed this I did not feel clever. I felt suspicious. The two things lined up too well, and in my experience a coincidence that clean is usually a result you have not finished explaining.

Where the analogy stops being a vibe

In 2014 Pankaj Mehta and David Schwab put a real object on the table: a paper titled An exact mapping between the Variational Renormalization Group and Deep Learning (arXiv 1410.3831). They constructed an explicit correspondence between one formulation of the renormalization group and a stacked Restricted Boltzmann Machine, and showed that training the network was, in their setup, performing a variational version of the coarse-graining flow. It was a clean demonstration that the analogy is not only poetic. Under the right restrictions, it is the same math.

But read it carefully and the honesty of the result is also its limit. The mapping holds for a specific, restricted class of energy-based models. It does not say that the ResNet on your GPU is secretly integrating out spins, or that backprop on cross-entropy is a renormalization flow in any literal sense. Later work pushed back, pointing out that the precise correspondence depends on assumptions you cannot carry over to arbitrary architectures, and that real networks build features in ways the renormalization group does not obviously license. The clean statement is narrow. The broad statement is a metaphor with good posture.

So I am left holding both. The analogy is too strong to be a coincidence and too loose to be a law.

The gap is where the work is

That gap is where the work is. If deep learning were provably renormalization, you could import decades of physics intuition wholesale: scaling, fixed points, relevant and irrelevant operators, and a principled answer to why these models generalize as well as they do. If it were obviously unrelated, fine, file the resemblance under pareidolia and move on. It is neither, and the unresolved part is a list of real questions. What is the analog of the rescaling step when there is no spatial lattice, only a sequence of tokens? When two very different architectures land on similar internal features, is that universality with a mechanism, or just two roads to the same dataset?

I do not know. I am wary of anyone who says they do, in either direction, because the confident versions of this story always seem to skip the part where you check.

I keep a renormalization experiment in my lab for exactly this reason. You set a temperature, watch the spins, then block them down and watch the coarse system carry the same critical behavior even after you have thrown most of it away. It is the cleanest demonstration I know that forgetting can be a form of understanding. Watching it run, I keep thinking the same thought about my day job, that the most important thing a learning system does might be the same thing physics does. Decide what does not matter, and let it go.

I run a lab of interactive physics on this site, and the renormalization toy is the one I reach for when a network's features start looking a little too familiar.