RuntimeError: NCCL Error 10: cuda malloc failed

Hm, haven’t seen this before. There’s a related thread here:

But maybe there the issue is due to some other problem. When i save models, I usually convert them to CPU first, because that keeps them most compatible. Then, when I reload them, I can put them on any GPU that are currently available.

Maybe you didn’t do that when you saved the model and when you try to load it, the original (or maybe not all) of the cuda devices (or the main cuda device) are not available?