just to add to @ptrblck’s answer, which version of pytorch are you using? There was a bug in <= 0.4.1 versions that pytorch uses small amount of memory in GPU-0 when you do not use torch.cuda.set_device() or CUDA_VISIBLE_DEVICES=x. If you are using version >= 0.5.0, this bug is fixed I guess.