Sparse Tensors in PyTorch

Ilya_Kostrikov · March 4, 2017, 3:39am

What is the current state of sparse tensors in PyTorch?

smth · March 4, 2017, 4:19am

right now, the description / summary of this PR that was merged 11 hours ago gives a good idea of the current state of things:

apaszke · March 4, 2017, 11:07am

But we’re not documenting them on purpose, because they might undergo some more changes in the future. The first step was to implement sprase updates for Embedding. Can I ask what’s your use case?

Ilya_Kostrikov · March 7, 2017, 11:45am

I need just basic sparse matrix multiplication in order to implement a Graph ConvNet model. For example:

and

github.com

mdeff/cnn_graph/blob/master/lib/models.py#L898


L = tf.SparseTensor(indices, L.data, L.shape)
L = tf.sparse_reorder(L)
# Transform to Chebyshev basis
x0 = tf.transpose(x, perm=[1, 2, 0])  # M x Fin x N
x0 = tf.reshape(x0, [M, Fin*N])  # M x Fin*N
x = tf.expand_dims(x0, 0)  # 1 x M x Fin*N
def concat(x, x_):
    x_ = tf.expand_dims(x_, 0)  # 1 x M x Fin*N
    return tf.concat([x, x_], axis=0)  # K x M x Fin*N
if K > 1:
    x1 = tf.sparse_tensor_dense_matmul(L, x0)
    x = concat(x, x1)
for k in range(2, K):
    x2 = 2 * tf.sparse_tensor_dense_matmul(L, x1) - x0  # M x Fin*N
    x = concat(x, x2)
    x0, x1 = x1, x2
x = tf.reshape(x, [K, M, Fin, N])  # K x M x Fin x N
x = tf.transpose(x, perm=[3,1,2,0])  # N x M x Fin x K
x = tf.reshape(x, [N*M, Fin*K])  # N*M x Fin*K
# Filter: Fin*Fout filters of order K, i.e. one filterbank per feature pair.
W = self._weight_variable([Fin*K, Fout], regularization=False)

The Laplacian matrix is extremely sparse is this case.

apaszke · March 7, 2017, 12:32pm

You need sparse x sparse -> sparse multiplication, right? Right now we only have sparse x dense -> dense and sparse x dense -> sparse, because that’s what we needed for sparse Embedding updates. You can open a feature request if you want.

Ilya_Kostrikov · March 7, 2017, 12:53pm

I need sparse x dense -> dense. So I can use PyTorch in this case. Thanks a lot!

apaszke · March 7, 2017, 1:16pm

torch.mm and torch.spmm should work then

Ilya_Kostrikov · March 7, 2017, 5:23pm

But it’s not in autograd yet?

Is there a fast way to add it?

smth · March 7, 2017, 7:32pm

a fast and local way is for you to write an autograd function for yourself.

hconroy · September 21, 2017, 3:08pm

That’s my question too (now on 21st Sept). Can anyone comment on the current state of sparse tensors in PyTorch?

Thank you

11131 · October 24, 2017, 8:14am

I would like to update variable with sparse gradients. and it’s normal. I know that wasn’t support by tensorflow. so how about pytorch/. Thank you!

pradeepgopalakrishna · December 4, 2018, 2:53pm

What is the status of Sparse Support in PyTorch? I’d like to contribute to the module. What are the current design strategies in place? There seems to be no detail or comprehensive discussion on this aspect.

rajeshpiryani · September 13, 2021, 6:01pm

https://pytorch.org/docs/stable/sparse.html#sparse-coo-tensors