Gradient w.r.t each sample

Using model.[layer_name].weight.grad.data I am able to access the gradients for weights of a particular layer. However, I assume that these gradients are averaged across samples in a mini-batch. Is there a way for me to access the gradients of weights for each sample?

I was able to obtain per-sample gradients for activations or neurons using register_hook, so not sure what to do about weights.

Thanks!

2 Likes

there is no way to access the gradients wrt weight for each individual sample. Gradients wrt weights in pytorch are always accumulated (even over the mini-batch). If you want gradients wrt each sample, you will have to run each sample individually through the network. This is because both the THNN backend and CuDNN dont support individual sample gradients wrt weight.