How to ensure my GPU is utilized to the fullest?

[TinfoilHat0]

Hi,

I’m monitoring my GPU usage as I’m doing training.
What I observe is, it fluctuates between 93% and 44%. Is this because, at some points, it waits CPU to pass data? Or is it because I do a pass over validation set (which is considerably smaller than the training set) after training ?

In any case, does these fluctuations mean I’m under-utilizing my GPU? If that’s the case, what I am doing wrong? For example, I tried to increase num_workers on my data loaders but it didnt have any effect on the usage pattern.

[wolterlw (Volodymyr)]

Check out these threads

Getting low GPU util with High GPU memory - How to correctly use DataLoader?

Since your dataset is tiny, I don’t think that multiple workers will help you much. It seems you are just slicing the tensors currently without any transformations. You could try to load all data, push it to the GPU beforehand and just slice the batch manually in your training loop. Maybe this will speedup your model a bit. However, your data and model might be just too small to get a high GPU utilization. As a small side note: you shouldn’t call the forward method of your model, but the mod…

Gpu utilization is less

Dataloader creates a new PROCESS for every worker, and the dataset has to be “copied” to every worker. Depending on whether you’re on Windows or Linux (spawning a process in Windows is much more expensive than forking one in Linux), and how the dataset stores it’s data (Tensors don’t seem to get copied from what I’ve tested, but Python structures, yes), you might have a very high overhead for creating the processes. Unless you’re working with some supercomputer, I believe 8 workers is more than…

93% is excellent utilization and I believe having lower GPU utilization during validation is expected as you don’t compute gradients or make parameter updates so the process is a lot more data-intensive
I’d suggest profiling your DataLoader vs the training step as the first step to figure out where the bottleneck actually is.

[TinfoilHat0]

Thanks for the answer. So, it makes sense to GPU usage to dip during validation for the reasons you specified. In that case, since training and validation are independent, is there a way to force GPU to start the training of next epoch while doing validation of the previous epoch ?