Dense x Sparse matrix multiplcation producing invalid combination of arguments in addmm

1

I’m trying to make a simple feed forward network with sparse weights. Using pytorch v0.1.12, python 3 [cuda80] I am unable to do a simple forward pass in the two ways shown below, both produce the same error.

The shape of x is [10,100] and the sparse tensor w1 is of shape [100,500].
I expect a dense tensor of shape [10,500].

x = Variable(torch.from_numpy(p_in).type(dtype), requires_grad=False)

#where weights[0] is of type torch.sparse.FloatTensor
w1 = Variable(weights[0].type(dtype_sparse), requires_grad=True)

#this is where the error occurs
l3 = torch.mm(x, w1)
#also used this:
l3 = x.mm(w1)

The output of this is:

TypeError Traceback (most recent call last)
in ()
----> 8 l3 = torch.mm(x, w1)

/home/lina/anaconda2/envs/py3/lib/python3.5/site-packages/torch/autograd/variable.py in mm(self, matrix)
525 def mm(self, matrix):
526 output = Variable(self.data.new(self.data.size(0), matrix.data.size(1)))
→ 527 return self._static_blas(Addmm, (output, 0, 1, self, matrix), False)
528
529 def bmm(self, batch):

/home/lina/anaconda2/envs/py3/lib/python3.5/site-packages/torch/autograd/variable.py in _static_blas(cls, args, inplace)
518 if num_args == 4:
519 alpha = args[1]
→ 520 return cls(alpha, beta, inplace)(*(args[:1] + args[-2:]))
521
522 def _blas(self, cls, args, inplace):

/home/lina/anaconda2/envs/py3/lib/python3.5/site-packages/torch/autograd/_functions/blas.py in forward(self, add_matrix, matrix1, matrix2)
26 output = self._get_output(add_matrix)
27 return torch.addmm(self.alpha, add_matrix, self.beta,
—> 28 matrix1, matrix2, out=output)
29
30 def backward(self, grad_output):

TypeError: torch.addmm received an invalid combination of arguments - got (int, torch.FloatTensor, int, torch.FloatTensor, torch.sparse.FloatTensor, out=torch.FloatTensor), but expected one of:

  • (torch.FloatTensor source, torch.FloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
  • (torch.FloatTensor source, torch.SparseFloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
  • (float beta, torch.FloatTensor source, torch.FloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
  • (torch.FloatTensor source, float alpha, torch.FloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
  • (float beta, torch.FloatTensor source, torch.SparseFloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
  • (torch.FloatTensor source, float alpha, torch.SparseFloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
  • (float beta, torch.FloatTensor source, float alpha, torch.FloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
    didn’t match because some of the arguments have invalid types: (int, torch.FloatTensor, int, torch.FloatTensor, torch.sparse.FloatTensor, out=torch.FloatTensor)
  • (float beta, torch.FloatTensor source, float alpha, torch.SparseFloatTensor mat1, torch.FloatTensor mat2, *, torch.FloatTensor out)
    didn’t match because some of the arguments have invalid types: (int, torch.FloatTensor, int, torch.FloatTensor, torch.sparse.FloatTensor, out=torch.FloatTensor)
2

Seems it is just not currently possible to do dense X sparse > dense (please correct me if wrong)

Workaround is to just have sparsity manually. I.e. make the “sparse” matrix dense, and multiplying by a mask of 1s and 0s each trial. Main goal here was to have a dense input layer mm by a sparse weight matrix to produce a dense next layer. Then of course to apply backprop to these sparse layers, and needed the sparse values to not be updated. By multiplying by the mask, those unwanted changes resets values to 0.