Calculate the gradients w.r.t non-zero values in a sparse matrix


I’m using a sparse matrix in part of my neural network and I was wondering if there is a way to get the gradients with respect to only the non zero values of a sparse matrix. If I convert the sparse matrix into a dense one, the matrix is going to be too large and it’s going to take too much time to compute the gradient so I want to keep my matrix sparse.

I would appreciate it if someone can help me with this.


Unfortunately, at the moment, very few of the backward formulas formulas support sparse gradients :confused:
But you can call autograd.grad(out, inp, sparse_grad_out) to try and propagate sparse gradient in the backward pass.
If you want to create them during the backward, I think a custom Function can be used to do the conversion you want.

Ok, Thanks for the help!