How to debug a detected anomoly

autograd
1

Hi, I have this error - I think it was raised because I have torch.autograd.set_anomoaly_detect(True).

[W python_anomaly_mode.cpp:104] Warning: Error detected in CudnnRnnBackward. Traceback of forward call that caused the error:
  File "/share/miniconda3/envs/cs21/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/share/miniconda3/envs/cs21/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/share/miniconda3/envs/cs21/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/share/usr/will/sw/Conv-TasNet/src/multi_conv_tasnet.py", line 58, in forward
    spatial_mixture = self.spatial_encoder(mixture)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/share/sw/Conv-TasNet/src/multi_conv_tasnet.py", line 132, in forward
    mixture_w, (self.h, self.c) = self.lstm(mixtures_l)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/share/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 581, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
 (function _print_stack)

I assumed it maybe a vanishing/exploding gradient issue but I am already using gradient clipping… Does anyone know how I could debug it?

Thanks in advance

1 Like