This code works fine in PyTorch 1.11.0 and 1.12.0
import torch print(torch.__version__) torch.use_deterministic_algorithms(True) t = torch.tensor(range(10), dtype=float) t = t.cuda() x = torch.cumsum(t, 0) print(x)
On PyTorch 1.13 it fails with error
RuntimeError: cumsum_cuda_kernel does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
I found an open issue about it Feature Request: deterministic CUDA cumsum · Issue #89492 · pytorch/pytorch · GitHub
We run our models with determinism and without it – both modes are used wildly on the same models.
I tried to write a simple workaround like this
def cumsum(*args: Any, **kwargs: Any) -> Any: # `cumsum_cuda_kernel` does not have a deterministic implementation # we turn off determinism enforcement just for this one operation, otherwise it error # in the deterministic mode. is_deterministic = torch.are_deterministic_algorithms_enabled() is_warn_only = False try: if is_deterministic: is_warn_only = torch.is_deterministic_algorithms_warn_only_enabled() torch.use_deterministic_algorithms(False) result = torch.cumsum(*args, **kwargs) finally: if is_deterministic: torch.use_deterministic_algorithms(True, warn_only=is_warn_only) return result
But then when we are trying to TorchScript the model we run into
E RuntimeError: E Unknown builtin op: aten::are_deterministic_algorithms_enabled. E Could not find any similar ops to aten::are_deterministic_algorithms_enabled. This op may not exist or may not be currently supported in TorchScript.
Is there any recommended way to go about it?