Machine lags while training with Cuda

As soon as I start training a model with gpu/cuda, my machine begins to lag. I’m on Ubuntu 16.04. It lags to the point where I cannot effectively use Firefox or Sublime Text, and I’m at a loss for how to troubleshoot this. Any suggestions?

while training model with GPU/CUDA we are also using CPU for driving the GPU and running data augmentation.

If you must use the same machine whilst training, either improve the hardware or sacrifice the training speed.

As for the latter, monitor your resources with htop. If you lack memory, either turn down DataLoader's num_workers or batch_size. If you lack CPU cores, turn down DataLoader's num_workers only.

You can also monitor disk IO with atop; if you have a single disk and it’s 100% busy, use h5py to zip your data in prior. Or, get a new disk and seperate your OS from your data.