Function "'UnsafeViewBackward'" returned nan values in its 0th output

1

[W python_anomaly_mode.cpp:104] Warning: Error detected in UnsafeViewBackward. Traceback of forward call that caused the error:
File “/anaconda3/lib/python3.6/threading.py”, line 884, in _bootstrap
self._bootstrap_inner()
File “/anaconda3/lib/python3.6/threading.py”, line 916, in _bootstrap_inner
self.run()
File “/anaconda3/lib/python3.6/threading.py”, line 864, in run
self._target(*self._args, **self._kwargs)
File “/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker
output = module(*input, **kwargs)
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
result = self.forward(*input, **kwargs)
File “”, line 166, in forward
pixels = pixel_attn(pixels) + pixels
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
result = self.forward(*input, **kwargs)
File “”, line 31, in forward
return self.fn(self.norm(x), **kwargs)
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
result = self.forward(*input, **kwargs)
File “”, line 69, in forward
q, k, v = self.to_qkv(x).chunk(3, dim = -1)
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
result = self.forward(*input, **kwargs)
File “/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py”, line 93, in forward
return F.linear(input, self.weight, self.bias)
File “/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py”, line 1692, in linear
output = input.matmul(weight.t())
(function _print_stack)

who can help me, i feel very terrible

2

i meet many backward errors, i check the inputs do not contain nan

3

I assume self.to_qkv uses a linear layer internally? If so, could you check the activation inputs as well as the parameters for invalid values right before the error is raised?

4

when i use single GPU, no error, but 3 GPU, many erros like “***Backward nan”