You can use PyTorch for years without ever having to write a backward pass. Then someone asks you for one in plain NumPy, on a two-layer network, and the chain rule you know on paper turns into a fight with matrix shapes. Here's how I'd write it, along with the trick that'll keep those shapes straight for you.
We'll take a batch X of shape (N, D), one hidden layer of width H with ReLU, and C output classes:
W1 is (D, H), b1 is (H,), W2 is (H, C), b2 is (C,). Broadcasting adds the bias to all of the rows. None of this needs a loop over examples, and if your interview answer's got one, you'll be asked to take it out.
Write G_X for ∂L/∂X, which always has the same shape as X. We start at the loss and walk back through the forward pass a line at a time:
Source: DEV - interview — Read the full article on the original site ↗