Alternating Cuda errors: "invalid device" and "CUDNN_STATUS_NOT_INITIALIZED"

You might be running into this error.
If that’s the case (PyTorch 1.8.0 pip wheels with CUDA10.2 runtime on a Turing GPU), then install the nightly release, the CUDA11.1 pip wheel, or any conda binary to fix it.