You probably should create a new thread with your issue, this thread is 2 years old.
Also how do you check memory usage. As far as I know, once you have allocated GPU memory, PyTorch keeps it even when unused to quickly reuse it later without having to request it from Cuda, see Reserving gpu memory? so you can’t simply use nvidia-smi.