[Reporting bug] INTERNAL ASSERT FAILED at "C:/w/b/windows/pytorch/aten/src\\ATen/native/cuda/Reduce.cuh":929, please report a bug to PyTorch

C_J · April 22, 2022, 5:59am

I have below error message:

~~~~\lib\site-packages\torch\autograd\__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: sub_iter.strides(0)[0] == 0INTERNAL ASSERT FAILED at "C:/w/b/windows/pytorch/aten/src\\ATen/native/cuda/Reduce.cuh":929, please report a bug to PyTorch.

I have this issue when I increase my model size (not batch size).
I have checked that this is not a memory issue, it only uses half of the free memory at its maximum.

Thanks for reading!

ptrblck · April 22, 2022, 6:13am

Are you seeing this issue using the latest master build or nightly binary? If so, could you post a minimal, executable code snippet to reproduce the issue, please?

C_J · April 22, 2022, 6:26am

Thanks for reply,
I really want to, but my code very complicated (or messy) at this point so I’m not sure I could do that.

But what I can say is, this is happening alternatively with another memory issue.

A (normal case). Normal model size, batch size=2 per each GPU (takes <50% of total gpu memory)
B (CUDA out of memory). Normal model size, batch size=3or2 per each GPU(takes <<70% of total gpu memory) link: [CUDA out of memory] How to reserve memory in GPU?
C (This case). Larger model size, batch size=1 per each GPU (takes <50% of total gpu memory)

So I think this is something related to the memory issue, but I have no clue.

Thanks,

C_J · April 22, 2022, 6:28am

I’m using pytorch 1.10 at window 10 machine.
I’m sorry but I don’t know what you mean by master build or nightly binary.

ptrblck · April 22, 2022, 7:10am

By “master build” I was referring to a source build from the current master branch on GitHub and by “nightly binary” I was referring to the pip wheels or conda binaries, which are built each night from the current master branch. You can install the latter binaries by selecting “Preview (Nightly)” here.

H_A · January 1, 2023, 5:58am

I am having the same issue and using Google Colab’s GPU.
the error is occuring on the line of code is:
import torch

RuntimeError Traceback (most recent call last)
in
2 #If the video is corrupted delete the video.
3 import glob
----> 4 import torch
5 import torchvision
6 from torchvision import transforms

/usr/local/lib/python3.8/dist-packages/torch/init.py in
190 if USE_GLOBAL_DEPS:
191 _load_global_deps()
→ 192 from torch._C import * # noqa: F403
193
194 # Appease the type checker; ordinarily this binding is inserted by the

RuntimeError: THPDtypeType.tp_dict == nullptr INTERNAL ASSERT FAILED at “…/torch/csrc/Dtype.cpp”:135, please report a bug to PyTorch.

Alexander_Goryunov · January 3, 2023, 10:13pm

I am in the same boat. Using Google Colab with torch version 1.13.0+cu116

 6 import yaml
      7 from sklearn.model_selection import train_test_split
----> 8 from torch.utils.data import Subset
      9 from torch.nn.modules.container import ModuleList
     10 from transformers import AutoTokenizer, TrainingArguments

/usr/local/lib/python3.8/dist-packages/torch/__init__.py in <module>
    190     if USE_GLOBAL_DEPS:
    191         _load_global_deps()
--> 192     from torch._C import *  # noqa: F403
    193 
    194 # Appease the type checker; ordinarily this binding is inserted by the

RuntimeError: THPDtypeType.tp_dict == nullptr INTERNAL ASSERT FAILED at "../torch/csrc/Dtype.cpp":135, please report a bug to PyTorch.

And trying to import torch:

ValueError                                Traceback (most recent call last)

<ipython-input-6-beb55f24d5ec> in <module>
----> 1 import torch
      2 print(torch.__version__)

/usr/local/lib/python3.8/dist-packages/torch/__init__.py in <module>
    190     if USE_GLOBAL_DEPS:
    191         _load_global_deps()
--> 192     from torch._C import *  # noqa: F403
    193 
    194 # Appease the type checker; ordinarily this binding is inserted by the

ValueError: module functions cannot set METH_CLASS or METH_STATIC

Restarting the environment made the errors magically disappear.

ptrblck · January 3, 2023, 11:19pm

I cannot reproduce the issue in a new Colab notebook using torch==1.13.0+cu116 and sklearn==1.0.2.

Alexander_Goryunov · January 3, 2023, 11:21pm

It probably has something to do with Colab itself, because I don’t see it anymore either.

msf · January 17, 2023, 10:23pm

Hi all,
I have this error too, but every time I restart the runtime in google Colab, the error is fixed, and after running the code two times, I again have this error. Thanks for the help
false INTERNAL ASSERT FAILED at “…/c10/cuda/CUDAGraphsC10Utils.h”:73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus32740

ptrblck · January 18, 2023, 1:43am

Could you share a minimal and executable code snippet to reproduce the issue as well as the output of python -m torch.utils.collect_env, please?

Ruicky8 · February 2, 2023, 8:17am

Hi!

I was having this error on colab today, and as mary said I fixed it restarting the runtime and changing from CPU to GPU. I have run the command python -m torch.utils.collect_env before and after the runtime restart, maybe it’s useful for you.

Before the runtime restart

Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.6
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.147+-x86_64-with-glibc2.29
Is CUDA available: False
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.1
[pip3] torchvision==0.14.1+cu116
[conda] Could not collect

After the restart:

Collecting environment information...
PyTorch version: 1.13.1+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.6
Libc version: glibc-2.31

Python version: 3.8.10 (default, Nov 14 2022, 12:59:47)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.147+-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.13.1+cu116
[pip3] torchaudio==0.13.1+cu116
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.1
[pip3] torchvision==0.14.1+cu116
[conda] Could not collect

Also I was encountering the error when running the following:

import torch
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"