Backpropagation Calculus: How Neural Networks Actually Learn

01 The Problem Learning Has to Solve

A neural network is, at its core, a function with a great many knobs. Each connection between neurons carries a weight, and the network's behavior is entirely determined by the setting of those weights. When the network is first initialized, those weights are random, and the output is garbage. Learning is the process of nudging every weight in the direction that makes the output less wrong. The question is how, given a network with billions of weights, you decide which way to nudge each one.

The answer is backpropagation, and the mathematical machinery underneath it is nothing more exotic than the chain rule from introductory calculus. What makes the chain rule powerful here is that a deep network is a composition of simple functions, one per layer. To know how a weight in the first layer affects the final error, you multiply the derivatives of every layer that sits between them. Backpropagation is the algorithm that computes all of those products efficiently, in a single backward sweep, so that training a billion-parameter model is feasible at all.

02 Forward Pass: Building the Thing to Measure

Before anything can be corrected, the network has to produce an answer. In the forward pass, input data flows through each layer: every neuron takes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function. The output of one layer becomes the input to the next, until the final layer emits a prediction. That prediction is compared against the correct answer using a loss function, which returns a single number measuring how badly the network did on this example.

The choice of loss function matters because it has to be differentiable. Common cross-entropy loss for classification and mean squared error for regression are both smooth enough that their gradients exist everywhere the network operates. The loss is the quantity the entire learning process is trying to reduce, and its gradient, the vector of partial derivatives with respect to every weight, is the compass that points downhill. The forward pass exists to compute the loss; the backward pass exists to compute its gradient.

03 The Chain Rule, Layer by Layer

The mathematical heart of backpropagation is the observation that a composite function's derivative is the product of the derivatives of its parts. If the network is a stack of layers, each layer's contribution to the final error depends on how much the next layer amplified or suppressed the signal coming back toward it. Working from the output backward, the algorithm first computes the gradient of the loss with respect to the final layer's activations, then uses that to compute the gradient with respect to the final layer's weights, and simultaneously passes a signal back to the previous layer.

This backward flow is where the name comes from. Each layer receives an error signal, multiplies it by the local derivative of its own activation, and forwards a new error signal to the layer below. By the time the sweep reaches the input, every weight in the network has been assigned a gradient that says, in effect, how much it contributed to the mistake and which way it should move to fix it. The cost of the whole backward pass is comparable to one forward pass, which is why training is described as roughly twice the cost of inference.

Figure 1. Illustrative gradient magnitudes across an eleven-layer network. Vanishing gradients decay toward early layers; exploding gradients grow; healthy propagation stays near unity.

04 Why Depth Makes It Hard

The chain rule is elegant in a single layer and painful across many. When you multiply together dozens of derivatives, two failure modes appear. If the per-layer derivatives are mostly less than one, their product collapses toward zero as it travels back through the layers, so the earliest layers of a deep network receive almost no learning signal. This is the vanishing gradient problem, and it is the reason plain deep networks were once thought untrainable. If the derivatives are mostly greater than one, the product balloons, the early-layer gradients become enormous, and training diverges. This is the exploding gradient problem.

The history of deep learning is, in large part, a sequence of engineering fixes for these two failure modes. Rectified linear activations, which avoid saturating to zero, batch normalization, which keeps activations on a controlled scale, residual connections, which give the gradient a shortcut path around layers, and careful initialization all exist to keep the backward signal alive across depth. Each of these techniques is, at bottom, a way of keeping the product of derivatives from collapsing or exploding. They are the reason a hundred-layer transformer can be trained at all.

05 The Learning Rate: Where Calculus Meets Control

Once gradients are computed, the network updates its weights by moving a small step in the opposite direction of each gradient. The size of that step is the learning rate, and it is the single most consequential hyperparameter in training. Too large and the network overshoots the minimum of the loss, bouncing around or diverging; too small and training crawls, possibly stopping before it reaches a good solution. Selecting and adapting the learning rate is where the clean mathematics of the chain rule meets the messier practice of optimization.

Figure 2. Training loss curves under three learning-rate regimes. Only the well-tuned rate reaches a low loss; the others plateau or diverge.

Modern optimizers do not use a single fixed rate. Adam and its variants maintain a per-parameter learning rate adapted from the running statistics of the gradient, and learning-rate schedules decay the global rate over training so that the network takes large exploratory steps early and small refining steps late. The combination turns a brittle knob into a self-correcting system, but the underlying operation is still the same: take the gradient the chain rule produced, scale it by a rate, and subtract it from the weights.

06 From Calculus to the Models You Use

Every contemporary model, from the smallest open-weight release to the largest frontier system, is trained by this same backward sweep. The transformer architecture that underlies GPT, Gemini, Claude, and the open models is, for backpropagation's purposes, just another composition of differentiable layers. Attention is a weighted average with softmax weights, feed-forward blocks are matrix multiplications and nonlinearities, and every one of those operations has a known local derivative. The chain rule threads through all of them.

This is why backpropagation is the most load-bearing idea in modern machine learning. It is not a clever trick bolted onto the side of deep learning; it is the mechanism that makes deep learning tractable. The scale of the models has grown by many orders of magnitude, the architectures have changed repeatedly, and the hardware has been rebuilt around the operation, but the core computation in every training step is still the same backward pass described decades ago. The frontier moved; the engine did not.

07 The History and the Open Questions

The algorithm has a longer history than its recent fame suggests. The core idea was described by Paul Werbos in his 1974 doctoral work, independently rediscovered by David Rumelhart, Geoffrey Hinton, and Ronald Williams in the 1980s, and then sat largely dormant until compute and data caught up with it in the 2010s. The mathematical content did not change during those decades; what changed was the ability to run it at scale. That gap between an idea and its viability is a recurring pattern in the field.

Open questions remain even as the algorithm dominates. Second-order methods, which use curvature information to take smarter steps than plain gradients, are theoretically attractive but practically expensive at scale, and the tradeoff between first-order speed and second-order accuracy is still active research. The geometry of the loss landscapes these methods traverse, and why gradient descent finds good solutions in such high-dimensional spaces at all, is only partially understood. Backpropagation works; explaining precisely why it works so well is harder than running it.