Backpropagation is repeated application of the chain rule. This lab turns that abstraction into two visible signals: activations carry a prediction forward; gradients carry responsibility for the error backward.
Cyan pulses are forward activations. Pink pulses are backward gradients. Node brightness shows magnitude; dim early layers reveal a vanishing signal.
Each layer performs a local operation: multiply by a weight, add a bias, then apply a nonlinear activation. The forward pass stores intermediate values. The backward pass reuses them to calculate local derivatives and multiply sensitivity through the graph.
The loss does not depend on an early weight directly. It depends on the weight through every downstream activation. The chain rule converts that nested dependency into a product of local slopes.
Store pre-activations and outputs. Without them, derivatives such as the ReLU gate or sigmoid slope cannot be evaluated for this example.
Each node receives an upstream gradient, multiplies by its local derivative, and sends the result toward earlier parameters.
Gradient descent subtracts learning-rate times gradient. Backprop computes direction; the optimizer decides how to move.
Sigmoid and tanh flatten at large magnitudes. Their derivatives approach zero, so multiplying many saturated slopes erases the signal before it reaches early layers.
sigmoid'(z) = a(1-a)
maximum slope = 0.25A ReLU passes positive values and maps negative values to zero. When a unit remains negative, its local derivative is zero and no gradient reaches its incoming weight for that example.
ReLU'(z) = 1 if z > 0
ReLU'(z) = 0 otherwiseRepeated multiplication by weights or Jacobians with magnitudes above one can amplify gradients. Clipping, careful initialization, normalization, residual paths, and gated architectures help control scale.
gradient norm > threshold
scale gradient before updateKeeping the phases separate makes debugging much easier. A wrong prediction, invalid derivative, and unstable optimizer update produce different symptoms and require different fixes.
Feed the input into the first layer. Every layer computes its weighted pre-activation and nonlinear output. The final activation is the prediction.
z[l] = w[l] * a[l-1] + b[l] a[l] = activation(z[l])
Compare prediction with target. This lab uses squared error so the first backward signal is prediction minus target.
L = 0.5 * (prediction - target)^2 dL/dprediction = prediction - target
Move from output to input. Multiply the upstream gradient by activation slope, then use the result for weight, bias, and previous-activation gradients.
delta[l] = upstream * activation'(z[l]) dL/dw[l] = delta[l] * a[l-1]
Apply the optimizer step only after all gradients are computed. Updating a weight during backward traversal would make later gradients refer to a different network.
w[l] = w[l] - learning_rate * dL/dw[l] b[l] = b[l] - learning_rate * delta[l]
No. It is reverse-mode automatic differentiation on a computational graph. Neural networks are a prominent use because they contain many parameters but usually produce one scalar loss.
Shared intermediate calculations would be repeated. Reverse mode accumulates downstream sensitivity once and reuses it, making it efficient when many parameters influence one output.
No. The gradient is a local direction of steepest increase. An optimizer transforms it using a learning rate, momentum, adaptive scaling, clipping, weight decay, or other rules.
They create short identity paths through which activations and gradients can travel, reducing reliance on long products of nonlinear derivatives.
Explore SuperPowers workflows that decompose goals into visible agent steps, whether the task is research, computer use, or producing a complete website.
Explore SuperPowersWebsite-building agentComputer-use cache