Physics-Informed Neural Networks

Most of machine learning learns a function from data and nothing else. A physics-informed neural network learns a function from data and from the differential equation that function is supposed to obey. You take the governing equation, move everything to one side so the true solution makes it zero, and add that residual to the loss. The network is then punished twice: once for disagreeing with your measurements, and once for disagreeing with physics.

I gave a talk on this at TU München in early 2025, walking through the theory and then building three solvers from scratch. This is the writeup of that talk. The slides, with all the plots, are linked at the end.

The idea: put the equation in the loss

A PINN is an ordinary neural network $\hat{u}(x, t; \theta)$ trained to satisfy a differential equation, not just to fit points. The whole trick lives in the loss, a weighted sum of two terms:

$\mathcal{L}_\text{total} = \lambda_1\,\mathcal{L}_\text{physics} + \lambda_2\,\mathcal{L}_\text{data}.$

The data loss is ordinary supervised error against whatever you actually know: measurements, initial values, boundary values.

$\mathcal{L}_\text{data} = \frac{1}{N_d} \sum_{i=1}^{N_d} \left| u(x_i, t_i) - \hat{u}(x_i, t_i) \right|^2.$

The physics loss is the part that earns the name. Write the equation as $\mathcal{F}(u) = 0$, so $\mathcal{F}$ is its residual: plug the true solution in and you get zero. Now scatter a cloud of collocation points across the domain, evaluate the residual on the network's prediction at each one, and penalise it for being nonzero.

$\mathcal{L}_\text{physics} = \frac{1}{N_r} \sum_{i=1}^{N_r} \left| \mathcal{F}\big(\hat{u}(x_i, t_i)\big) \right|^2.$

The derivatives inside $\mathcal{F}$ come from automatic differentiation, the same machinery that computes gradients for training, turned on the inputs instead of the weights. No mesh, no finite differences: you get the equation's derivatives exactly, anywhere you ask. And collocation points are effectively free, since they carry no labels, only coordinates, so you can enforce the physics across the whole domain even where you have no data at all.

Application I: a damped harmonic oscillator

The cleanest place to start is an ODE with a closed form you can check against. The under-damped harmonic oscillator, $\delta < \omega_0$ with $\delta = \mu / 2m$ and $\omega_0 = \sqrt{k/m}$:

$m\,\frac{d^2 x}{dt^2} + \mu\,\frac{dx}{dt} + k\,x = 0, \qquad x(0) = 1,\quad \dot{x}(0) = 0.$

I took $m = 1$, $\mu = 4$, $k = 400$, so $\omega_0 = 20$ and the system rings while it decays. The loss has three pieces: the ODE residual on the interior collocation points, and the two initial conditions pinned at $t = 0$.

$\mathcal{L} = \frac{1}{N} \sum_i \big| L\hat{u}(t_i; \theta) \big|^2 + \big| \hat{u}(0; \theta) - x_0 \big|^2 + \big| \dot{\hat{u}}(0; \theta) - v_0 \big|^2.$

A small network was enough: four layers, two of them hidden with 32 units each, $\tanh$ activations, ADAM at a learning rate of $10^{-3}$ for 25k epochs. The prediction lands on top of the analytic solution across the whole interval, decaying oscillation and all, trained from two boundary points and a scatter of unlabelled collocation points.

Turning it around: recovering a parameter you do not know

The forward problem is the boring half. The reason PINNs are interesting is that the same loss runs backwards. Suppose you do not know the damping $\mu$, but you have some noisy measurements of the motion. Make $\mu$ a trainable parameter and hand it to the optimiser alongside the network weights:

$\mathcal{L}(\theta, \mu) = \frac{1}{N} \sum_i \big( L(\mu)\,\hat{u}(t_i; \theta) \big)^2 + \frac{\lambda}{M} \sum_j \big( \hat{u}(t_j; \theta) - u_\text{obs}(t_j) \big)^2.$

In code this is almost anticlimactic, torch.optim.Adam(list(pinn.parameters()) + [mu], lr=1e-3). The network learns the trajectory that fits the noisy data while obeying the ODE, and to do that it has to settle on the value of $\mu$ that makes both terms small. Over training the estimate climbs and converges to the theoretical $\mu = 2\delta = 4$. The physics term acts as a regulariser strong enough to pull a physical constant back out of noisy data, using nothing but the same architecture with one extra scalar in the parameter list.

Application II: the heat equation

From an ODE to a PDE. The 1D heat equation,

$\frac{\partial u}{\partial t} = \alpha\,\frac{\partial^2 u}{\partial x^2}, \qquad (x, t) \in [0, L] \times [0, T], \quad \alpha = 0.01,$

with a cosine initial profile $u(x, 0) = \cos(5\pi x)$ and insulating Neumann boundaries, $\partial u / \partial x = 0$ at both ends. Neumann conditions are a nice stress test, because they constrain a derivative at the boundary, which for a PINN is just one more autodiff residual to drop into the loss, with no special treatment. A slightly larger network handled it: four hidden layers of 64 units, $\tanh$, 10k epochs, with 5000 interior collocation points plus 1000 on the boundaries and 1000 on the initial line. The absolute error against the analytic solution stays small across space and time, worst where the gradients are sharpest, exactly where you would expect.

Two dimensions, and where it starts to strain

The 2D heat equation is the same idea with one more Laplacian term,

$\frac{\partial u}{\partial t} = \alpha\left( \frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2} \right), \qquad u(x, y, 0) = \cos(3\pi x)\cos(3\pi y),$

again with Neumann boundaries on every side. Here I switched the activation to SiLU and added a learning-rate schedule (decay $\gamma = 0.9$ every 1000 steps), with 5000 collocation points and 5000 more each on the boundaries and the initial slice.

The interesting part was measuring where it fails. Instead of collapsing everything into one error number, I took the 2D Fourier transform of the error field,

$e(x, y) = u_\text{num} - u_\text{exact}, \qquad \hat{E}(k_x, k_y) = \mathcal{F}\{e(x, y)\},$

to see which spatial frequencies the network was getting wrong. That is a far more honest diagnostic than a mean error: it shows the model smoothing over the high-frequency structure rather than failing uniformly, which points straight at the fix. A network carries a spectral bias toward low frequencies, and the error spectrum is where you catch it. More collocation points, another layer or two, or wider hidden layers, 128 units instead of 64, all buy the detail back.

What I took from it

PINNs are not a free lunch. Training is finicky, the loss weights $\lambda$ genuinely matter, and a stiff or high-frequency problem will punish an undersized network. But the core move, that you can write a physical law as a residual and make a network respect it, is real, and the inverse problem is the part that sold me: recovering a physical constant from noisy data by insisting the answer obey the equation. That is the physicist's instinct, refusing a fit you cannot justify, written down as a loss function.

These were slides I presented at TU München on 29 January 2025. The full deck, with every plot and result, is here: Physics-Informed Neural Networks (PINNs): Theory and Applications in Computational Physics.

Cheers.