I’m trying to train a simple LSTM model. The input size is about 11KB (same as target size). When I run the training loop with a batch size of 32, everything seems to work well. CPU usage is under 10% (even though that feels a bit excessive) and GPU usage is around 30%…
When I increase the batch size to 48, the CPU usage goes to 100% and everything slows down (training takes twice as long). GPU usage is virtually unchanged at around 30%. I checked with batch size of 40 and that seems fine. At 48 something seems to change and triggers the constant CPU usage. Everything else seems ok (plenty of RAM left, hardly any disk activity, etc).
Any suggestions or ideas on how to troubleshoot this? At the batch size of 48 the total input size should be about 528KB. Not sure if this triggers some kind of threshold somewhere. Any help is greatly appreciated.