I implemented the approach here and then if I figured out how many batches each process (GPU) would get and take the minimum, restricted any process from forward/backward any batches in an epoch over that amount…all the hangs went away!
I implemented the approach here and then if I figured out how many batches each process (GPU) would get and take the minimum, restricted any process from forward/backward any batches in an epoch over that amount…all the hangs went away!