Detect anomaly in C++

I’m having -nan loss in my training, so I’ve looked up in this forum for a solution. I’ve found this discussion where the Anomaly Detection module is recommended. But I’m working in C++, and couldn’t found if it’s available there, or any documentation. If someone has an example or documentation, I’ll appreciate it.
Thanks in advance.

Have you tried torch::autograd::AnomalyMode::set_enabled(true); from torch/csrc/autograd/anomaly_mode.h?

Best regards

Thomas

I tried that, but I’d forgotten to add the include path. Thank you Thomas.
Anyway, now the forward pass it’s throwing me a SIGSEGV, don’t know why.

Hello,

Just to let you know that it does not work for me neither.

I had the Exception in thread pool task: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDAFloatType [122, 512]], which is output 0 of SelectBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Every time I enable AnomalyMode with v1.5, I have a SIGSEGV in my module when I call x.to(device). I do not have enough time to debug this right now and it is not a critical issue on my side. I will take a look into it as soon as I will have time but I would like to know if this is working for someone.

Thank you

Pascal