[Context]
Book: Deep Learning from Scratch
Jupyter Notebook on GitHub, Code block 55
[Question]
Why element-wise multiplication is applied to calculate dLdN = dLdS*dSdN, rather than matrix multiplication via either np.dot() or np.matmul()?
I assume this is to make the dimensionality of the rest derivatives correct as shown below in the comment following each derivative. However, I still don’t understand why the calculation of dLdN is different from that of dLdX… Or, am I missing something important here?
def matrix_function_backward_sum_1(X: ndarray,
W: ndarray,
sigma: Array_Function) -> ndarray:
'''
Compute derivative of matrix function with a sum with respect to the
first matrix input
'''
assert X.shape[1] == W.shape[0] # X: (m x n), W: (n x p)
# matrix multiplication
N = np.dot(X, W) # N: (m x p)
# feeding the output of the matrix multiplication through sigma
S = sigma(N) # S: (m x p)
# sum all the elements
L = np.sum(S) # L: a scalar
# note: I'll refer to the derivatives by their quantities here,
# unlike the math where we referred to their function names
# dLdS - just 1s
dLdS = np.ones_like(S) # (m x p)
# dSdN
dSdN = deriv(sigma, N) # (m x p)
# dLdN
dLdN = dLdS * dSdN # (m x p) element-wise multiplication?
# dNdX
dNdX = np.transpose(W, (1, 0)) # (p x n)
# dLdX
dLdX = np.dot(dSdN, dNdX) # (m x p) x (p x n) = (m x n)
return dLdX