I am a beginner trying to learn PyTorch and there is one question bugging me. I know PyTorch support sparse x dense -> dense function in torch.mm. However, I don’t think it currently supports autograd on sparse variables (say sparse matrix). Examples are:
x = torch.sparse.FloatTensor(2,10)
y = torch.FloatTensor(10, 5)
sx = torch.autograd.Variable(x)
sy = torch.autograd.Variable(y)
torch.mm(sx, sy) # fails
Errors: TypeError: Type torch.sparse.FloatTensor doesn’t implement stateless method addmm
Am I on the wrong track?
Thanks for any comment!
you guys should use
torch.matmul which is implemented for both sparse and dense.
torch.matmul for sparse*dense matrix multiplication but get the
torch.sparse.FloatTensor doesn't implement stateless method addmm error from both.
it does indeed look like it isn’t working. i’m looking into it.
So the bummer is, autograd is not supported for sparse matrices. Sorry for not looking closely when i replied last.
We dont have any tests in test_autograd for sparse either.
We used sparse directly in nn layers such as
nn.Embedding at the implementation level.
We’re tracking the finishing of this feature here: https://github.com/pytorch/pytorch/issues/2389
torch.mm in particular, it wasn’t that hard for me to make it work with autograd. you can probably patch these in while I send a PR with unit tests and all: https://gist.github.com/anonymous/49c10bc17ac4a97307d52c07d01a2870
Thanks Soumith for looking into this. Patched the changes and it looks like
torch.mm works on an example I tried.
torch.matmul works on a dot b where a is sparse tensor, and b has to have dimension > 1.
So far until 0.2, sparse tensor can’t be the second operand, otherwise there will be some error like
"invalid combination of arguments".
(torch.SparseDoubleTensor source, torch.DoubleTensor mat2) is what is expected.
In order to perform autograd, you may define a dot product inheriting
torch.autograd.Function. And also squeeze or unsqueeze the dimension when necessary.
Don’t know when this will be resolved.
This patch was working fine with SGD. Today I tried Adadelta and RMSprop but got the following error :
'torch.cuda.sparse.FloatTensor' object has no attribute 'addcmul_'
I am curious if the PR @smth mentioned is there yet??
So far I am on version 0.2.0+5de7f9e, and it seems the
torch.mm still didnt support the sparse x dense -> dense ?
I also raised some discussion in https://github.com/pytorch/pytorch/issues/2389#issuecomment-342119147
Just curious about what is the fastest way to make this work?
Or it just I found the wrong function to use?
thx a lot guys!
I cannot find the file torch/autograd/_functions/blas.py for PyTorch 0.4.0.
Also the variable.py does not contain the stuff in the patch.
So how can we make PyTorch 0.4.0 mm support autograd on Sparse Tensor?
mm works for sparse x dense -> dense, but calling backward() on the result leads to error:
RuntimeError Traceback (most recent call last)
<ipython-input-47-12626f95b38d> in <module>()
----> 1 s.backward()
th04/lib/python3.5/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
91 products. Defaults to ``False``.
---> 93 torch.autograd.backward(self, gradient, retain_graph, create_graph)
95 def register_hook(self, hook):
th04/lib/python3.5/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
88 tensors, grad_tensors, retain_graph, create_graph,
---> 89 allow_unreachable=True) # allow_unreachable flag
RuntimeError: Expected object of type torch.FloatTensor but found type torch.sparse.FloatTensor for argument #2 'mat2'
How was this resolved? I still get an ‘Expected object of backend CPu but got backedn SparseCPU for argument #2 ‘mat2’’ error