Terminate called after throwing an instance of 'c10::Error' what() : CUDA error: uncorrectable NVLink error detected during the execution

1

When running faster_rcnn,I found this error,I hope someone can help me!
I just runned epoch0

image
image1920×573 107 KB

2

Could you rerun the script with:

NCCL_DEBUG=INFO TORCH_DISTRIBUTED_DEBUG=DETAIL python script.py args

and post the logs here, please?

3

Sorry, I can’t run these scripts because I use a high computing platform to calculate,

image
image743×60 4.2 KB

4

Can I chat with you privately?

5

script.py and args are only placeholders and you should replace them with your actual script file name and its arguments.

6

thanks,I’ve solved it

7

How did you resolve the issue and what was the root cause?