[Reporting bug] INTERNAL ASSERT FAILED at "C:/w/b/windows/pytorch/aten/src\\ATen/native/cuda/Reduce.cuh":929, please report a bug to PyTorch

I have below error message:

~~~~\lib\site-packages\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: sub_iter.strides(0)[0] == 0INTERNAL ASSERT FAILED at "C:/w/b/windows/pytorch/aten/src\\ATen/native/cuda/Reduce.cuh":929, please report a bug to PyTorch.

I have this issue when I increase my model size (not batch size).
I have checked that this is not a memory issue, it only uses half of the free memory at its maximum.

Thanks for reading!

Are you seeing this issue using the latest master build or nightly binary? If so, could you post a minimal, executable code snippet to reproduce the issue, please?

Thanks for reply,
I really want to, but my code very complicated (or messy) at this point so I’m not sure I could do that.

But what I can say is, this is happening alternatively with another memory issue.

A (normal case). Normal model size, batch size=2 per each GPU (takes <50% of total gpu memory)
B (CUDA out of memory). Normal model size, batch size=3or2 per each GPU(takes <<70% of total gpu memory) link: [CUDA out of memory] How to reserve memory in GPU?
C (This case). Larger model size, batch size=1 per each GPU (takes <50% of total gpu memory)

So I think this is something related to the memory issue, but I have no clue.

Thanks,

I’m using pytorch 1.10 at window 10 machine.
I’m sorry but I don’t know what you mean by master build or nightly binary.

By “master build” I was referring to a source build from the current master branch on GitHub and by “nightly binary” I was referring to the pip wheels or conda binaries, which are built each night from the current master branch. You can install the latter binaries by selecting “Preview (Nightly)” here.