# Volatile GPU util 0% with high memory usage **URL:** \ **Category:** vision\ **Created:** [July 30, 2019, 12:37pm UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047 "2019-07-30T12:37:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Fahad\_Jahangir](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fahad_jahangir/32/14587_2.png) [@Fahad\_Jahangir](https://discuss.pytorch.org/u/Fahad_Jahangir)\ **Post date:** [July 30, 2019, 12:37pm UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/1 "2019-07-30T12:37:35Z")
I have found this question posted on this forum for the multiple times with no working answer. I am trying to train a SegNet on satellite images using single GPU (Nvidia Tesla-k80 12GB). Memory-Usage is high but the volatile GPU-Util is 0%. ![prob](https://discuss.pytorch.org/uploads/default/original/2X/0/00c7304cd71a2aabd0d074de2ed259b49ce7272e.png) In the DataLoader, I have tried increasing the **num\_workers** , setting the **pin\_memory= True** , and **removed all the preprocessing** like Data Augmentation, Caching etc… but the problem is still persistent. The code works just fine if I change the dataset. Any help will be highly appreciated. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 30, 2019, 10:01pm UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/2 "2019-07-30T22:01:57Z")
> [@Fahad\_Jahangir](#): > > The code works just fine if I change the dataset. What did you change regarding the dataset? If the samples are smaller or you can preload the other dataset completely, it points to a data loading bottleneck using your current dataset. ---
**Author:** ![Aviv\_Shamsian](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aviv_shamsian/32/12377_2.png) [@Aviv\_Shamsian](https://discuss.pytorch.org/u/Aviv_Shamsian)\ **Post date:** [July 31, 2019, 7:08am UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/3 "2019-07-31T07:08:41Z")
Hi Patrick, What are the general guidelines in order to make optimal GPU and CPU usage. I faced the same problem mentioned here when I tried to run the 'training classifier" notebook tutorial. When I changed the batch size nothing changed in the GPU usage (~600-700MB). The setup: CIFAR10 dataset 2 X 1080 ti (nvidia) 8 processors 32GB RAM memory ---
**Author:** ![Fahad\_Jahangir](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fahad_jahangir/32/14587_2.png) [@Fahad\_Jahangir](https://discuss.pytorch.org/u/Fahad_Jahangir)\ **Post date:** [July 31, 2019, 12:22pm UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/4 "2019-07-31T12:22:03Z")
Yes, the other dataset have smaller samples (images with low spatial resolution). How can I avoid this bottleneck situation? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 31, 2019, 2:12pm UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/5 "2019-07-31T14:12:42Z")
@rwightman shared some insights on data bottlenecks [here](https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19). CC @Aviv_Shamsian ---
**Author:** ![Aviv\_Shamsian](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aviv_shamsian/32/12377_2.png) [@Aviv\_Shamsian](https://discuss.pytorch.org/u/Aviv_Shamsian)\ **Post date:** [August 1, 2019, 6:00am UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/6 "2019-08-01T06:00:52Z")
Hi @ptrblck Thank you for your quick response! Can you please address to my specific problem? I read the comment you tagged me in but I have done all is written there. I’m using the Pytorch Dataloader (without any unnecessary for loops). How can I be sure what is the main cause of this problem? When I use `htop` in terminal all the six processors assigned to the mission are loaded with work but the GPU stays in ~600-700MB capacity regardless to the batch size (which increased to 2048 at the most). This is the reason why I suspect that the preprocess part is the bottleneck in this example. Just to mention no augmentations where preformed only reading, normalizing and converting to tensor. ---
**Author:** ![Fahad\_Jahangir](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fahad_jahangir/32/14587_2.png) [@Fahad\_Jahangir](https://discuss.pytorch.org/u/Fahad_Jahangir)\ **Post date:** [August 1, 2019, 9:55am UTC](https://discuss.pytorch.org/t/volatile-gpu-util-0-with-high-memory-usage/52047/8 "2019-08-01T09:55:27Z")
Hi @ptrblck Thanks for you response. I realized that the sample size was very large and took so long to load, So I tried cashing the data, it takes more memory which I can afford however once the data was cached, Volatile GPU-Util reached maximum.