The papers that rewired me

A paper earns a place on this list by one test only: it changed how I think, not how impressed I was. Impressive is cheap. Half the famous results I admire didn't move a single belief; I read them, nodded, and went back to whatever I already thought. The ones below did damage. They left me unable to look at a learning system the same way again. That is the bar.

The Bitter Lesson

Sutton's claim is short enough to put on a sticky note: over seventy years, general methods that ride raw compute have beaten the clever priors we hand-engineer, every single time. The first time I read it I disagreed, hard, because I came from physics, where the prior is the point. A symmetry, a conservation law, a clean Hamiltonian: those are the crown jewels, and Sutton was telling me they lose. He was mostly right and I hated it. What it rewired was my reflex to encode what I know. Now I ask first whether the structure I want to impose is a real invariant or just a comfort.

Attention Is All You Need

The transformer threw out recurrence and convolution and bet everything on attention, and the bet paid off so completely that the whole field reorganized around it. I don't keep this one for the architecture. I keep it for the lesson in its title, which read like arrogance in 2017 and reads like prophecy now. A small group looked at the accumulated machinery of sequence modeling and asked which parts were load-bearing, and the answer was that most of it wasn't, which is a much harder thing to discover than a new layer to bolt on.

An exact mapping between the Variational Renormalization Group and Deep Learning

Mehta and Schwab showed that stacking restricted Boltzmann machines is, under a clean construction, the same operation as a renormalization group transformation in physics: coarse-grain, throw away the irrelevant, keep what survives. I found this paper at exactly the moment I needed permission to take my own intuition seriously. Feature learning as coarse-graining was not a metaphor I was forcing onto the data. Someone had written the equality down. I have a longer piece on that collision over in my writing, and this is where it started.

Deep Learning and the Information Bottleneck Principle

Tishby and Zaslavsky framed training as a tradeoff: compress the input as much as you can while keeping the bits that predict the output. The information bottleneck gave me a vocabulary for a question I'd been asking badly, namely what a network actually throws away and why throwing things away is the whole job. The empirical story around it has been argued about since, and that's fine. The frame survived the arguments for me. Learning is lossy on purpose.

Deep Double Descent

Nakkiran and coauthors documented a curve that, by the textbook I grew up on, should not exist: push a model past the point where it perfectly fits the training data and test error goes down again instead of exploding. The first time I saw the double descent plot I assumed it was a bug in someone's pipeline. It wasn't. It was reality filing a bug report against my mental model of overfitting, and reality wins those.

Probability Theory: The Logic of Science

Jaynes argues that probability is not about frequencies or coins but about reasoning under incomplete information, an extension of logic itself, and he argues it with a physicist's contempt for hand-waving. The book rebuilt my Bayesianism from the floor up. Before it, I used priors because the formula had a slot for them. After it, I understood a prior as an honest statement of what I knew before the data arrived, and a posterior as what I'm allowed to believe after. The difference sounds small, and yet I have not made a modeling decision since without it sitting somewhere underneath the choice.

A Mathematical Theory of Communication

Shannon, in 1948, gave information a unit and proved you could push it through a noisy channel at a definite rate and no faster. I list it last because it underwrites everything above it. Entropy, mutual information, the bottleneck, the very idea that a model "captures bits" about the world: all of it descends from one paper that decided meaning was a distraction and counting was the point. Almost everything I think about learning is, when you dig, a footnote to that decision.

These are not the best papers in machine learning. They are the ones that left a mark on me, and the mark is the only credential I trust. If your list looks like mine, we probably broke in the same places.

I poke at these instincts in public over at my lab, where the experiments are allowed to embarrass me before production does.