As soon as I start training a model with gpu/cuda, my machine begins to lag. I’m on Ubuntu 16.04. It lags to the point where I cannot effectively use Firefox or Sublime Text, and I’m at a loss for how to troubleshoot this. Any suggestions?
while training model with GPU/CUDA we are also using CPU for driving the GPU and running data augmentation.
If you must use the same machine whilst training, either improve the hardware or sacrifice the training speed.
As for the latter, monitor your resources with
htop. If you lack memory, either turn down
batch_size. If you lack CPU cores, turn down
You can also monitor disk IO with
atop; if you have a single disk and it’s 100% busy, use
h5py to zip your data in prior. Or, get a new disk and seperate your OS from your data.