CUDA error from trying to load dataset with tqdm?

1

tqdm is just a visual progress bar helper.

for i, (x, y) in enumerate(tqdm(self.mnist_dataset)):
       ...

That enumerate tqdm portion triggers this error:

File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 34, in _pin_memory_loop
    data = pin_memory(data, device)                                            
  File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 65, in pin_memory                                                                            
    return type(data)([pin_memory(sample, device) for sample in data])  # type: ignore[call-arg]                                                                                                       
  File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 65, in <listcomp>         
    return type(data)([pin_memory(sample, device) for sample in data])  # type: ignore[call-arg]
  File "/u/me/conda/envs/env/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 50, in pin_memory                                                                             
    return data.pin_memory(device)                                                                                                                                                                     
RuntimeError: CUDA error: out of memory
2

Is your code working fine without the usage of tqdm? If so, could you post a minimal, executable code snippet to reproduce the issue, please?

4

Extremely minimal example:

from torch.utils.data import DataLoader
from torchvision.datasets.mnist import MNIST
from torchvision.transforms import ToTensor


dataset = MNIST('./', download=True, transform=ToTensor())

dataset = DataLoader(dataset=dataset,
                     pin_memory=True)  # pin_memory triggers CUDA error

for _ in dataset:
    continue

It seems the pin_memory=true is responsible. Running on school’s A6000 GPUs.

5

Thanks for the code snippet. I cannot reproduce it using 1.14.0.dev20221207+cu117 so you might need to update to recent version in case you are using an older release.