Is it more efficient to load all tensors to GPU first or do it batch wise?

[Abhai_Kollara]

Is it better to call .cuda() on my input and tensors first and store it first or should I do that in my batch generator. Does it make a difference ?

[SnowWalkerJ]

I think so, as long as it fits in your GPU.
If you do it each batch, the time spent on transferring data from memory to GPU can be relatively large. That’s why you may observe a low GPU-Util with nvidia-smi.

[Abhai_Kollara]

Thank you for the reply. @SnowWalkerJ

My thoughts were the same, but I’m running into an issue with DataLoader returning non-CUDA tensors, which made me think that that’s not how it was meant to be used. Have you faced this issue ?

Dataloader returning non-CUDA Tensors

I create a simple dataset that has two CUDA tensors, one of shape (1000, 5) and the other (1000,) but when I load them using a DataLoader only the first one is returned as CUDA tensor. I think this may have something to do with the default collate_fn of the DataLoader. Any fixes ? import torch from torch.utils.data import Dataset, DataLoader import numpy as np x = torch.cuda.FloatTensor(np.random.randn(1000, 5)) y = torch.cuda.LongTensor(np.random.randint(0,10, size=(1000, ))) class data(Data…