Hi everyone, first post here and not sure if this is the right category . Mods please move my post to the right category if this is the wrong one.
I mainly train 3D CNN models and I recently updated from PyTorch 1.5 to 1.10. While trying to train my model on the updated environment, I encountered this error:
RuntimeError: avg_pool3d_backward_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if thats acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation
I’ve trained non-deterministic models in the past but the main limitation of 3D medical imaging datasets is that they’re very small, and are thus prone to large fluctuations in performance based on the random seed. I found that there are large fluctuations even with a fixed torch.manual-seed().
I’d really appreciate if support for MaxPool3d and AvgPool3d could be prioritized as (I hope) refactoring MaxPool2d/AvgPool2d determinism for these operations shouldn’t be very difficult.
Thanks dev team for the great work as always