The common way is to start your code with:
use_cuda = torch.cuda.is_available()
Then, each time you create a new instance of any tensor/variable/module, just do:
if use_cuda:
my_obect.cuda()
That way you make sure that everything is stored or not on GPU or CPU (by default, without calling .cuda() it will be on CPU)