Backpropagation
Gradient descent needs to know how much each weight contributed to the error — but a deep network has millions of weights buried behind other weights. Backpropagation answers it with the chain rule, applied backwards: start with the error at the output, then pass blame layer by layer toward the input, each node splitting its share among the weights that fed it, in proportion to how much they carried. One backward sweep computes every gradient at once, which is what makes training deep networks feasible at all.
Run the forward pass, then step the backward pass one layer at a time. Watch the error appear at the output and fan out backwards — every node ends up holding its own share of the blame.
Forward pass: the input flows left to right and produces a prediction — which is wrong by 0.8.
How it works
Blame is the everyday word; the mathematical object is a partial derivative — how much the loss would change if this weight nudged slightly. The chain rule says derivatives through composed functions multiply, so each backward step is a multiplication by the local weights and the activation's slope. This is also where deep networks get sick: multiply many numbers smaller than one and the signal fades to nothing before reaching early layers (vanishing gradients); many numbers larger than one and it blows up (exploding gradients). Architectures you meet later — residual connections, normalization, careful initialization, the LSTM's gates — are largely engineering to keep this backward signal healthy.
Check yourself
Why must the blame at a hidden node be computed after the blame at every node in the layer to its right?
Go deeper (free): 3Blue1Brown — Backpropagation, intuitively ↗