I am training an image segmentation model and the time taken per each step in epoch seems to be rising throughput whole epoch. I am not sure if it has something to do with the way I am feeding my model with images. I am using DataLoader utility. I have pasted time for one epoch. Is there anyway to i…

Are you only seeing this slowdown using nn.DataParallel or also a single GPU? In the latter case, are you also seeing an increased memory usage? If that’s the case, you might accidentally store some tensors which are still attached to the computation graph, or somehow extend the computation graph …

Slow training with nn.DataParallel

vision

samra-irshad (Samra Irshad) August 3, 2020, 7:05am 3

Thanks I was saving some tensors in loop.

Why the training slow down with time if training continuously? And Gpu utilization begins to jitter dramatically?