Does PyTorch support autograd on sparse matrix?

autograd
1

Hi there,

I am a beginner trying to learn PyTorch and there is one question bugging me. I know PyTorch support sparse x dense -> dense function in torch.mm. However, I don’t think it currently supports autograd on sparse variables (say sparse matrix). Examples are:

x = torch.sparse.FloatTensor(2,10)
y = torch.FloatTensor(10, 5)
sx = torch.autograd.Variable(x)
sy = torch.autograd.Variable(y)

torch.mm(sx, sy) # fails

Errors: TypeError: Type torch.sparse.FloatTensor doesn’t implement stateless method addmm

Am I on the wrong track?

Thanks for any comment!

Yiru

2 Likes
2

Same question! Any idea?

3

you guys should use torch.matmul which is implemented for both sparse and dense.

4

I tried torch.mm and torch.matmul for sparse*dense matrix multiplication but get the torch.sparse.FloatTensor doesn't implement stateless method addmm error from both.

5

it does indeed look like it isn’t working. i’m looking into it.

6

So the bummer is, autograd is not supported for sparse matrices. Sorry for not looking closely when i replied last.

We dont have any tests in test_autograd for sparse either.

We used sparse directly in nn layers such as nn.Embedding at the implementation level.

7

We’re tracking the finishing of this feature here: https://github.com/pytorch/pytorch/issues/2389

For torch.mm in particular, it wasn’t that hard for me to make it work with autograd. you can probably patch these in while I send a PR with unit tests and all: https://gist.github.com/anonymous/49c10bc17ac4a97307d52c07d01a2870

4 Likes
8

Thanks Soumith for looking into this. Patched the changes and it looks like torch.mm works on an example I tried.

9

I tried, torch.matmul works on a dot b where a is sparse tensor, and b has to have dimension > 1.
So far until 0.2, sparse tensor can’t be the second operand, otherwise there will be some error like "invalid combination of arguments".
(torch.SparseDoubleTensor source, torch.DoubleTensor mat2) is what is expected.

In order to perform autograd, you may define a dot product inheriting torch.autograd.Function. And also squeeze or unsqueeze the dimension when necessary.

Don’t know when this will be resolved.

10

This patch was working fine with SGD. Today I tried Adadelta and RMSprop but got the following error : 'torch.cuda.sparse.FloatTensor' object has no attribute 'addcmul_'

11

I am curious if the PR @smth mentioned is there yet??
So far I am on version 0.2.0+5de7f9e, and it seems the torch.mm still didnt support the sparse x dense -> dense ?
I also raised some discussion in https://github.com/pytorch/pytorch/issues/2389#issuecomment-342119147
Just curious about what is the fastest way to make this work?
Or it just I found the wrong function to use?

thx a lot guys!

12

I cannot find the file torch/autograd/_functions/blas.py for PyTorch 0.4.0.
Also the variable.py does not contain the stuff in the patch.

So how can we make PyTorch 0.4.0 mm support autograd on Sparse Tensor?

mm works for sparse x dense -> dense, but calling backward() on the result leads to error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-47-12626f95b38d> in <module>()
----> 1 s.backward()

th04/lib/python3.5/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

th04/lib/python3.5/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag
     90
     91

RuntimeError: Expected object of type torch.FloatTensor but found type torch.sparse.FloatTensor for argument #2 'mat2'
14

Never mind, I find https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py, which solves the problem!

15

How was this resolved? I still get an ‘Expected object of backend CPu but got backedn SparseCPU for argument #2 ‘mat2’’ error