Pytorch backward error: sparse tensor

autograd
1

ask for help: I was using pytorch to train a GAN-based model. During the training process, I need to use a sparse tensor, which is calculated by the learnable parameters of the generator, which means that when I update the generator, I need to pass a sparse tensor with gradients to the discriminator, i.e. required_grads=True.

However, when doing backward() on the loss of the generator:

loss_g = generator_loss(pred_label, pred_real, pseudo_label, pseudo_train, is_real)
loss_g.backward()

I have the following problem:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2021.2\plugins\python\helpers\pydev\pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2021.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:/myprojects/Mywork/experiment_logs/2022_11_5/gan_demo.py", line 185, in <module>
    loss_g.backward()
  File "D:\anaconda3\envs\myenv\lib\site-packages\torch\_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "D:\anaconda3\envs\myenv\lib\site-packages\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: sparse tensors do not have strides

I tried to use the torchviz tool to view the computation graph of the generator loss, and the results are as follows:

Digraph.gv
Digraph.gv1864×6112 111 KB

2

what is the shape of your input tensor?

3

pred_label: shape=[2735, 7], dense_tensor, grad_fn=SparseAddmmBackward
pred_real: shape=[2735,2], dense_tensor, grad_fn=SparseAddmmBackward
pseudo_label: shape=[2735], dense_tensor, no grad ( i.e. each sample’s class)
pseudo_train: shape=[2735], dense_tensor, no grad ( i.e. train idx )
is_real: shape=[2735], dense_tensor, no grad (i.e. is real or generated)

Here is my generator_loss function:

def generator_loss(_pred_label, _pred_real, _pseudo_label, _pseudo_train, _is_real, loss_fn=nn.CrossEntropyLoss()):
    pseudo_idx = [ _ for _ in range(_is_real.nonzero()[-1]+1, _is_real.shape[0])]
    num_pseudo = len(pseudo_idx)
    loss_class = loss_fn(_pred_label[pseudo_idx].squeeze(), _pseudo_label[pseudo_idx].squeeze())
    _is_real_rev = torch.tensor([1]*num_pseudo).squeeze()
    loss_real = loss_fn(_pred_real[pseudo_idx].squeeze(), _is_real_rev)

    _loss_g = loss_real + loss_class
    print(f"---gen_loss:{_loss_g.item():.4f}")
    return _loss_g

Thank you for your reply.

4

I use a dense tensor and a sparse tensor to compute pred_label and pred_real:

pred_label, pred_real = discriminator.forward(pseudo_x, pseudo_adj)
loss_g = generator_loss(pred_label, pred_real, pseudo_label, pseudo_train, is_real)

The forward process of discriminator:

def forward(self, x, adj):
    x = self.conv1(x, adj)
    x = self.relu(x)
    x = self.dropout(x)
    pred_label = self.conv2(x, adj)
    pred_real = self.conv3(x, adj)

    return pred_label, pred_real

and conv layer is as follows:

def forward(self, x, adjacency):
    x = self.linear(x)
    x = torch.sparse.mm(adjacency, x)
    return x