In recent days, I have tried customizing functional conv2d. There are two confusions in my heart.
(1) Pytorch regards F.conv2d as a function (operation). So, it must have a forward and backward method.
But, I do not find them in the document. Where can I find them?
(2) The gradient of vector function is a Jacobi matrix. The output of F.conv2d is a matrix with high dimensions.
How does its gradient with respect to weight and bias look like (beyond my imagination)?
What the backward computes really is a vector Jacobian product, rather than the full Jacobian, for efficiency reasons, and also for the fact that most people really only care about the gradients of parameters w.r.t. a single scalar metric.
The current conv2d CPU backward is here https://github.com/pytorch/pytorch/blob/22e3b2c9c369c5fb44476eb538fa0a308df94eff/aten/src/THNN/generic/SpatialConvolutionMM.c#L247-L414