Everyone is told that neural networks learn by “backpropagation”, and everyone is then shown a wall of matrix equations bristling with transposes and tiny delta symbols. Let us throw that away and do something better : take one tiny network, put actual numbers through it, and watch the gradients being computed by hand. You will see that backpropagation is not an algorithm you need to memorise, it is just the chain rule walking backwards through a diagram, where each little box does one embarrassingly simple local calculation and whispers the result to its neighbour. By the end you will be able to backpropagate on paper.

1. A network is a graph of tiny operations

Forget “layers” for a moment. Any computation, including a neural network, can be drawn as a computational graph : a diagram where each node is one small operation (a multiply, an add, a squashing function) and arrows carry numbers from one node to the next.

Take the smallest possible “network” : a single neuron that multiplies its input $x$ by a weight $w$, adds a bias $b$, squashes the result with the sigmoid $\sigma(z) = \frac{1}{1+e^{-z}}$, and is scored against a target $y$ by the squared-error loss. As a graph, that is just five little boxes :

\[\begin{equation} x \xrightarrow{\;\times w\;} \; \xrightarrow{\;+\,b\;} z \xrightarrow{\;\sigma\;} a \xrightarrow{\;(a-y)^2\;} L. \label{eq:graph} \end{equation}\]

That is the whole model. Learning means answering one question : if I nudge $w$ a little, how does the loss $L$ change ? That number, $\frac{\partial L}{\partial w}$, is the gradient, and gradient descent will use it to improve $w$.

2. The forward pass : just compute

First we run the graph left to right with concrete numbers. Let us pick $x = 1$, $w = 0.5$, $b = 0$, and target $y = 1$. Nothing clever here, we simply evaluate each box and remember its output (we will need those outputs in a minute).

Figure 1 - The forward pass. Each node computes its output from its inputs : $wx = 0.5$, then $+b = 0.5 = z$, then $\sigma(z) = 0.622 = a$, then $(a-y)^2 = 0.143 = L$. We store every intermediate value.

3. The one rule of backpropagation

Now the only idea you need. The chain rule says that to get the gradient of $L$ with respect to something early in the graph, you multiply the local derivatives along the path connecting them. Backpropagation organises this multiplication cleverly : it flows a single number, “how much does $L$ change per unit change of my output ?”, backwards through the graph. Call the number arriving at a node its upstream gradient. Then every node obeys one rule :

upstream gradient into an input $=$ upstream gradient at the output $\times$ the node's own local derivative. $$ \frac{\partial L}{\partial(\text{input})} = \frac{\partial L}{\partial(\text{output})}\cdot\frac{\partial(\text{output})}{\partial(\text{input})}. $$

In plain words : each box is a little worker who knows only one thing, how sensitive its own output is to its own input. It takes the “blame” handed down from the right, scales it by that local sensitivity, and passes it further left. No box ever needs to understand the whole network. Here are the local derivatives our five boxes need, each a one-liner from calculus :

\[\begin{array}{ l c l } \text{loss } (a-y)^2 & : & \dfrac{\partial L}{\partial a} = 2(a - y) \\[4pt] \text{sigmoid } a=\sigma(z) & : & \dfrac{\partial a}{\partial z} = a(1-a) \\[4pt] \text{add } z = (wx) + b & : & \dfrac{\partial z}{\partial b} = 1,\quad \dfrac{\partial z}{\partial (wx)} = 1 \\[4pt] \text{multiply } wx & : & \dfrac{\partial (wx)}{\partial w} = x,\quad \dfrac{\partial (wx)}{\partial x} = w \end{array}\]

4. The backward pass : the same graph, in reverse

Start at the right with the trivial fact $\frac{\partial L}{\partial L} = 1$, and apply the rule box by box, reusing the values we stored during the forward pass.

\[\begin{aligned} \frac{\partial L}{\partial a} &= 2(a-y) = 2(0.622 - 1) = -0.755, \\ \frac{\partial L}{\partial z} &= \frac{\partial L}{\partial a}\cdot a(1-a) = -0.755 \times (0.622)(0.378) = -0.177, \\ \frac{\partial L}{\partial b} &= \frac{\partial L}{\partial z}\cdot 1 = -0.177, \\ \frac{\partial L}{\partial w} &= \frac{\partial L}{\partial z}\cdot x = -0.177 \times 1 = -0.177, \\ \frac{\partial L}{\partial x} &= \frac{\partial L}{\partial z}\cdot w = -0.177 \times 0.5 = -0.089. \end{aligned}\]

Figure 2 - The backward pass over the same graph. The orange number reaching each node is $\partial L / \partial(\text{that value})$. Each box multiplied the gradient handed to it by its own local derivative, precisely the backprop rule.

And that is backpropagation, complete, on a real example. We now know that increasing $w$ by a tiny amount decreases the loss (the gradient $-0.177$ is negative), so gradient descent will nudge $w$ up : $w \leftarrow w - \eta(-0.177)$. Do this repeatedly and the neuron learns.

Notice the fork at the multiply node : it sent gradient to both $w$ and $x$, each scaled by a different local derivative ($x$ and $w$ respectively). This is the general rule for a node with several inputs, give each input the upstream gradient times its own local derivative. When a value is used in several places, its gradients from those places simply add up.

5. Why do it backwards, and why it is fast

Why flow right-to-left instead of left-to-right ? Because there is one loss at the far right but many parameters at the left. Sweeping backwards from that single output computes the gradient for every parameter in one pass, sharing all the intermediate work. Computing it forwards would mean redoing the whole chain separately for each weight.

This sharing is the same trick as memoization : the upstream gradient at a node is computed once and reused for all the inputs feeding it. That is why the backward pass costs about the same as the forward pass, even for a network with billions of weights. Without it, training modern models would be flatly impossible. Under its formal name, reverse-mode automatic differentiation, this is exactly what PyTorch and TensorFlow do for you when you call .backward(). ¹

6. One honest warning : vanishing gradients

Look again at the backward pass : the gradient is a product of local derivatives, one per box on the path. The sigmoid’s local derivative $a(1-a)$ is at most $0.25$. Chain a dozen sigmoids and the gradient reaching the earliest layers is multiplied by $0.25$ a dozen times, roughly $0.25^{12} \approx 6\times10^{-8}$, effectively zero. Those early layers barely learn : the notorious vanishing gradient problem.

This single observation, that a long product of small numbers collapses, explains a surprising amount of modern design : the ReLU activation (local derivative exactly $1$ on the positive side, so the product does not shrink), careful weight initialisation, and the residual connections of the Transformer, which hand the gradient a clean shortcut straight back to the early layers.

7. Conclusion

Backpropagation deserves to feel easy, because it is : draw the computation as a graph, run it forwards once and remember the intermediate values, then sweep backwards multiplying by one local derivative at each box. The chain rule does all the work ; backprop is merely the bookkeeping that reuses shared results so the whole gradient falls out in a single pass. Feed those gradients to gradient descent and repeat a few million times, and the Universal Approximation Theorem’s promise, that a good network exists, is turned into a network you actually have.

References

1. Popularised for neural networks by Rumelhart, Hinton and Williams (Rumelhart et al., 1986) in 1986, though the reverse-mode differentiation at its core was known earlier, notably in Paul Werbos’s 1974 thesis. Every modern framework builds it in as automatic differentiation. ↩