# CUDA error: device-side assert triggered only for EMNIST dataset **URL:** \ **Category:** vision\ **Created:** [January 12, 2024, 8:49pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335 "2024-01-12T20:49:35Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![pytorchuser101](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorchuser101/32/63239_2.png) [@pytorchuser101](https://discuss.pytorch.org/u/pytorchuser101)\ **Post date:** [January 12, 2024, 8:49pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/1 "2024-01-12T20:49:35Z")
I get the CUDA device-side asset triggered only when running the model on Pytorch’s EMNIST dataset. It runs without any issues on MNIST, FashionMNIST, GTSRB, and Food101 I am changing the number of output neurons according to the dataset. The error message is as below ```auto ../aten/src/ATen/native/cuda/Loss.cu:240: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed. Traceback (most recent call last): File "train_t.py", line 351, in loss.backward() File "/home/dir/.local/lib/python3.8/site-packages/torch/_tensor.py", line 487, in backward torch.autograd.backward( File "/home/dir/.local/lib/python3.8/site-packages/torch/autograd/ __init__.py", line 200, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions ``` Version details: Torch : 2.0.1+cu117 Python : 3.8 Cuda : 11.7 Additional details: The models used are either pretrained on CIFAR10 or CIFAR100, So I use the following transforms when training on MNIST, FashionMNIST, or EMNIST: ```auto torchvision.transforms.Grayscale(num_output_channels=3), torchvision.transforms.Resize((32,32)) ``` Without the change in number of output channels, it throws the error : ```auto RuntimeError: output with shape [1, 32, 32] doesn't match the broadcast shape [3, 32, 32] ``` Any suggestion how to resolve the CUDA error for EMNIST ? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 12, 2024, 9:02pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/2 "2024-01-12T21:02:12Z")
Could you rerun the script with blocking launches as suggested in the error message and post the stacktrace here? ---
**Author:** ![pytorchuser101](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorchuser101/32/63239_2.png) [@pytorchuser101](https://discuss.pytorch.org/u/pytorchuser101)\ **Post date:** [January 12, 2024, 9:16pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/3 "2024-01-12T21:16:31Z")
This is with the launching block ```auto ../aten/src/ATen/native/cuda/Loss.cu:240: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [13,0,0] Assertion `t >= 0 && t < n_classes` failed. Traceback (most recent call last): File "train_t.py", line 350, in loss = criterion(outputs, y) File "/home/dir/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/dir/.local/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1174, in forward return F.cross_entropy(input, target, weight=self.weight, File "/home/dir/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 3029, in cross_entropy return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing) RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 12, 2024, 10:01pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/4 "2024-01-12T22:01:16Z")
```python Assertion `t >= 0 && t < n_classes` failed. ``` Your target contains values out of range so double check the output shape and the `min/max` values of your target. ---
**Author:** ![pytorchuser101](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorchuser101/32/63239_2.png) [@pytorchuser101](https://discuss.pytorch.org/u/pytorchuser101)\ **Post date:** [January 12, 2024, 10:14pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/5 "2024-01-12T22:14:08Z")
You are right, checked the output ranges. It is from [1-26] instead of [0-25] Any clue what the cause could be ? I am using Pytorch’s dataset as follows: ```auto trainset = torchvision.datasets.EMNIST(root='Datasets/EMNIST', split='letters', train=True, download=True) testset = torchvision.datasets.EMNIST(root='Datasets/EMNIST', split='letters', train=False, download=True) ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 12, 2024, 10:59pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/6 "2024-01-12T22:59:05Z")
Access the root folder of the dataset and check if e.g. an empty folder was created. ---
**Author:** ![pytorchuser101](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorchuser101/32/63239_2.png) [@pytorchuser101](https://discuss.pytorch.org/u/pytorchuser101)\ **Post date:** [January 13, 2024, 3:37pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/7 "2024-01-13T15:37:33Z")
I don’t see any empty folder or anything out of the ordinary. Traversing into EMNIST/EMNIST/raw, I see these files ```auto [dir@c32 raw]$ ls -l total 2277827 -rw-r--r-- 1 dir tsu 14739216 Jan 13 10:32 emnist-balanced-test-images-idx3-ubyte -rw-r--r-- 1 dir tsu 18808 Jan 13 10:33 emnist-balanced-test-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 88435216 Jan 13 10:33 emnist-balanced-train-images-idx3-ubyte -rw-r--r-- 1 dir tsu 112808 Jan 13 10:33 emnist-balanced-train-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 91197248 Jan 13 10:33 emnist-byclass-test-images-idx3-ubyte -rw-r--r-- 1 dir tsu 116331 Jan 13 10:33 emnist-byclass-test-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 547178704 Jan 13 10:33 emnist-byclass-train-images-idx3-ubyte -rw-r--r-- 1 dir tsu 697940 Jan 13 10:33 emnist-byclass-train-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 91197248 Jan 13 10:32 emnist-bymerge-test-images-idx3-ubyte -rw-r--r-- 1 dir tsu 116331 Jan 13 10:32 emnist-bymerge-test-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 547178704 Jan 13 10:33 emnist-bymerge-train-images-idx3-ubyte -rw-r--r-- 1 dir tsu 697940 Jan 13 10:32 emnist-bymerge-train-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 31360016 Jan 13 10:33 emnist-digits-test-images-idx3-ubyte -rw-r--r-- 1 dir tsu 40008 Jan 13 10:32 emnist-digits-test-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 188160016 Jan 13 10:33 emnist-digits-train-images-idx3-ubyte -rw-r--r-- 1 dir tsu 240008 Jan 13 10:33 emnist-digits-train-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 16307216 Jan 13 10:33 emnist-letters-test-images-idx3-ubyte -rw-r--r-- 1 dir tsu 20808 Jan 13 10:32 emnist-letters-test-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 97843216 Jan 13 10:33 emnist-letters-train-images-idx3-ubyte -rw-r--r-- 1 dir tsu 124808 Jan 13 10:33 emnist-letters-train-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 7840016 Jan 13 10:33 emnist-mnist-test-images-idx3-ubyte -rw-r--r-- 1 dir tsu 10008 Jan 13 10:33 emnist-mnist-test-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 47040016 Jan 13 10:32 emnist-mnist-train-images-idx3-ubyte -rw-r--r-- 1 dir tsu 60008 Jan 13 10:33 emnist-mnist-train-labels-idx1-ubyte -rw-r--r-- 1 dir tsu 561753746 Jan 13 10:32 gzip.zip ``` No other directories are present anywhere in the intermediate paths of EMNIST/EMNIST/raw ---
**Author:** ![pytorchuser101](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorchuser101/32/63239_2.png) [@pytorchuser101](https://discuss.pytorch.org/u/pytorchuser101)\ **Post date:** [January 15, 2024, 9:58pm UTC](https://discuss.pytorch.org/t/cuda-error-device-side-assert-triggered-only-for-emnist-dataset/195335/8 "2024-01-15T21:58:33Z")
If I print the actual labels of the trainset and testset of “letters” in EMNIST, I get the following values: [‘f’, ‘m’, ‘w’, ‘z’, ‘u’, ‘v’, ‘c’, ‘g’, ‘h’, ‘N/A’, ‘r’, ‘p’, ‘b’, ‘y’, ‘n’, ‘k’, ‘s’, ‘o’, ‘i’, ‘e’, ‘t’, ‘j’, ‘l’, ‘x’, ‘d’, ‘q’, ‘a’] The ‘N/A’ class changes the class count to 27, instead of 26. I redownloaded EMNIST, but still get the same error. All other splits of EMNIST - `byclass` , `bymerge` , `balanced` , `digits` and `mnist` work perfectly fine and have the accurate class counts.