RuntimeError: Function 'torch::autograd::CopySlices' returned nan values in its 0th output

Hi everyone,
I am getting the error in the title after 10 epochs of training.

If I run my code without Anomaly Detection, I get NaN’s in my data.

Running my code with Anomaly Detection enabled gives me the following output:

  File "/home/matteo/Code/PoseRefiner0/", line 90, in <module>
    def main(_run, n_epochs, learning_rate, training_batch_size, perform_node_updates, optimizer_function, use_lr_scheduler,
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/sacred/", line 190, in automain
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/sacred/", line 312, in run_commandline
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/sacred/", line 276, in run
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/sacred/", line 238, in __call__
    self.result = self.main_function(*args)
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/sacred/config/", line 42, in captured_function
    result = wrapped(*args, **kwargs)
  File "/home/matteo/Code/PoseRefiner0/", line 148, in main
    output = model(data)
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/torch/nn/modules/", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/matteo/Code/PoseRefiner0/", line 41, in forward
    x, edge_attr = self.msg_passer(x=x,                  # NOTE: This is different on purpose.
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/torch/nn/modules/", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/matteo/Code/PoseRefiner0/", line 49, in forward
    updated_edge_attr = self.update_edges(x=x, edge_index=edge_index, edge_attr=edge_attr,
  File "/home/matteo/Code/PoseRefiner0/", line 157, in update_edges
    updated_edge_attr[edge_id, :] = torch.mean(single_updates[ranges_for_averaging_EDGES[edge_id, ranges_for_averaging_EDGES[edge_id, :] != -1]], dim=0)
 (function print_stack)
ERROR - PoseRefiner-poses - Failed after 7:22:01!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/matteo/Code/PoseRefiner0/", line 158, in main
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/torch/", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/matteo/.conda/envs/PoseRefiner0/lib/python3.8/site-packages/torch/autograd/", line 125, in backward
RuntimeError: Function 'torch::autograd::CopySlices' returned nan values in its 0th output.

It seems that the following line is the culprit, and that while performing backpropagation something goes wrong:

updated_edge_attr[edge_id, :] = torch.mean(single_updates[ranges_for_averaging_EDGES[edge_id, ranges_for_averaging_EDGES[edge_id, :] != -1]], dim=0)

My hypotheses on what could be going wrong:

  • I might be using some incorrect values in my indices, possibly computing the average of an empty set. So then backprop would fail.
  • I might be computing an average of very large values. While summing the input values I could get an overflow, ruining the gradient.

Does anyone have a clue on what is happening or on how can I debug the problem further?

Thank you in advance!