Does PyTorch support autograd on sparse matrix?

(Yiru Shen) #1

Hi there,

I am a beginner trying to learn PyTorch and there is one question bugging me. I know PyTorch support sparse x dense -> dense function in However, I don’t think it currently supports autograd on sparse variables (say sparse matrix). Examples are:

x = torch.sparse.FloatTensor(2,10)
y = torch.FloatTensor(10, 5)
sx = torch.autograd.Variable(x)
sy = torch.autograd.Variable(y), sy) # fails

Errors: TypeError: Type torch.sparse.FloatTensor doesn’t implement stateless method addmm

Am I on the wrong track?

Thanks for any comment!


Sparse x Dense -> Dense matrix multiplication
(Nitish Gupta) #2

Same question! Any idea?


you guys should use torch.matmul which is implemented for both sparse and dense.

(Nitish Gupta) #4

I tried and torch.matmul for sparse*dense matrix multiplication but get the torch.sparse.FloatTensor doesn't implement stateless method addmm error from both.


it does indeed look like it isn’t working. i’m looking into it.


So the bummer is, autograd is not supported for sparse matrices. Sorry for not looking closely when i replied last.

We dont have any tests in test_autograd for sparse either.

We used sparse directly in nn layers such as nn.Embedding at the implementation level.


We’re tracking the finishing of this feature here:

For in particular, it wasn’t that hard for me to make it work with autograd. you can probably patch these in while I send a PR with unit tests and all:

(Nitish Gupta) #8

Thanks Soumith for looking into this. Patched the changes and it looks like works on an example I tried.


I tried, torch.matmul works on a dot b where a is sparse tensor, and b has to have dimension > 1.
So far until 0.2, sparse tensor can’t be the second operand, otherwise there will be some error like "invalid combination of arguments".
(torch.SparseDoubleTensor source, torch.DoubleTensor mat2) is what is expected.

In order to perform autograd, you may define a dot product inheriting torch.autograd.Function. And also squeeze or unsqueeze the dimension when necessary.

Don’t know when this will be resolved.

(Nitish Gupta) #10

This patch was working fine with SGD. Today I tried Adadelta and RMSprop but got the following error : 'torch.cuda.sparse.FloatTensor' object has no attribute 'addcmul_'


I am curious if the PR @smth mentioned is there yet??
So far I am on version 0.2.0+5de7f9e, and it seems the still didnt support the sparse x dense -> dense ?
I also raised some discussion in
Just curious about what is the fastest way to make this work?
Or it just I found the wrong function to use?

thx a lot guys!

(Laoreja) #12

I cannot find the file torch/autograd/_functions/ for PyTorch 0.4.0.
Also the does not contain the stuff in the patch.

So how can we make PyTorch 0.4.0 mm support autograd on Sparse Tensor?

mm works for sparse x dense -> dense, but calling backward() on the result leads to error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-47-12626f95b38d> in <module>()
----> 1 s.backward()

th04/lib/python3.5/site-packages/torch/ in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     95     def register_hook(self, hook):

th04/lib/python3.5/site-packages/torch/autograd/ in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Expected object of type torch.FloatTensor but found type torch.sparse.FloatTensor for argument #2 'mat2'

(Laoreja) #14

Never mind, I find, which solves the problem!