Torch.prod produces RuntimeError: CUDA driver error: invalid argument

Just had this happen to me with Python 3.10 in a virtualenv, system has cuda 12.2 installed, virtualenv is pytorch 2.0.1 cu118. Erasing the files in ~/.cache/torch/kernels solved it for me too.

1 Like

Cleaning cache works for me too. But looks like we need to delete that cache each time when we want to run something, otherwise the error still appears.I suppose with this approach we can’t run things in parallel as deleting the cache might impact different runs. Is there any official solutions yet? @ptrblck

In my case, cleaning the cache does not help. I solved this problem by manually assigning CUDA_HOME env variable (to a same version cuda folder as pytorch installed with).

Hope this can help in your case.

Issue solved after remove files under ~/.cache/torch/kernels/.

It’s weird, anybody could explain this code-wise ?