RunTime Error : tensors are on different GPUs

Even though I explicitly set the torch device to gpu-0 with the torch.cuda.set_device(0), I notice that my model loads in multiple GPUs.

ae = torch.load(MODEL_PATH).eval()

Is there a way to circumvent this issue?