Hi,
I had similar issue and spent several days to figure out.
In my case, I turned on torch.backend.cudnn.benchmark and it keeps evaluate on each batch which also causing memory fluctuating.
Hope this will help.
1 Like