This comment was the answer in my case when using pretrained language models.
When using provided tokenizers be sure to use the correct embedding size that fits with the model you are using.
It is less obvious as the error happens โout of nowhereโ, when the one document that has the longer embedding shows up. And also because the error is not the result of some error in the prediction/ training loop.
In my case it was mismatched tokenizer.pad_token_id, your model and tokenizer should use the same one 
Increase max_position_embeddings from its default value of 512
config = RobertaConfig(
max_position_embeddings = 1024,
)
model = RobertaForMaskedLM(config=config)
i think this is a problem because of vocab size the same problem was coming in my code snippent. The reason was that i updated the vocab with 2 extra token (โsosโ and โeosโ) but i forgot to increase the size of vocab
I notice that there still is a lot of confusion about this.
I had the same error for BertForTokenClassification. No problem on cpu and mps, but CUDA raised errors. Spent way too much time trying to solve. In my case, the error was probably due to the padding class labels (-100) and some mismatch with the tokenizer. This may have something to do with what @addisonklinke was referring to in this thread (January 2020). -100 should be the default index to be ignored by the model. By changing it to positive, e.g. +100 or even just 0 as one of the two existing class labels, I would get errors about the classes being in mismatch with num_labels (when defining the model).
I solved it by adding this line after defining the model and the tokenizer, thanks to pytorch - RuntimeError: CUDA error: device-side assert triggered - BART model - Stack Overflow and https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-when-fine-tune-bert/99170/7:
model.resize_token_embeddings(len(tokenizer))
Iโm seeing this issue as well though in my case I donโt think it is as simple as index out of bounds. My code looks like this
C, X = classwise_tensor.shape
logging.error(f"Expand info: CT shape: {classwise_tensor.shape} CT range [{torch.min(classwise_tensor)}. {torch.max(classwise_tensor)}] {classwise_tensor.dtype} "
f"Labels shape: {labels_flat.shape} label range [{torch.min(labels_flat)}, {torch.max(labels_flat)}] {labels_flat.dtype}")
assert 0 <= torch.min(labels_flat) and torch.max(labels_flat) < C
value_flat = torch.index_select(classwise_tensor, dim=0, index=labels_flat)
I run this line many many times so there are a lot of logs. Itโs a big script and not so easy to compress it into a reproducible size, but I think this is enough to say that itโs not just index out of bounds. torch version is โ2.2.2+cu121โ
Usually it crashes in the first few minutes of a run, this one happened after >40 steps (batch size 16, 8 GPUs). I have some runs with basically the same parameters that have survived for days though so Iโm not sure if it just tends to happen at the start more often or depends on what machine itโs allocated on or itโs itโs some user error Iโm not thinking of.
All the errors crash on the last line, There are some 11k lines of this logs before the crash, looking through them I donโt see anything suspicious. Note that I am not crashing on the assertion.
[20:47:28] ERROR Expand info: CT shape: torch.Size([11, 11]) CT range [0.0. confusion_error.py:58
1.0] torch.float32Labels shape: torch.Size([3145728])
label range [0, 9] torch.int64
Epoch 0: 11%|โ | 47/445 [03:25<29:01, 4.38s/it, v_num=mple]
Epoch 0: 11%|โ | 47/445 [03:25<29:01, 4.38s/it, v_num=mple][20:47:28] ERROR Expand info: CT shape: torch.Size([11, 11]) CT range [0.0. confusion_error.py:58
1.0] torch.float32Labels shape: torch.Size([4194304])
label range [0, 6] torch.int64
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1290: indexSelectLargeIndex: block: [313,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...this goes on for a while, stopping multiple chunks of 32 threads in blocks 313, 208, and 310,
โ /home/alex.li/git/JupiterCVML/kore/losses/utils/confusion_error.py:61 in โ
โ expand_classwise_to_image_shape โ
โ โ
โ 58 โ โ logging.error(f"Expand info: CT shape: {classwise_tensor.shape} CT range [{torch โ
โ 59 โ โ โ โ โ f"Labels shape: {labels_flat.shape} label range [{torch.min(labels โ
โ 60 โ โ assert 0 <= torch.min(labels_flat) and torch.max(labels_flat) < C โ
โ โฑ 61 โ โ value_flat = torch.index_select(classwise_tensor, dim=0, index=labels_flat) โ
โ 62 โ โ value = value_flat.reshape(B, H, W, X) โ
โ 63 โ โ # Transform ignore values from classwise_tensor[0,:] to 0 โ
โ 64 โ โ value[ignore_inds] = 0 โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
During handling of the above exception, another exception occurred:
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Traceback (most recent call last) โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ /home/alex.li/git/JupiterCVML/kore/scripts/train_seg.py:38 in <module> โ
โ โ
โ 35 โ
โ 36 โ
โ 37 if __name__ == "__main__": โ
โ โฑ 38 โ main() โ
โ 39 โ
โ โ
โ /home/alex.li/git/JupiterCVML/kore/scripts/train_seg.py:34 in main โ
โ โ
โ 31 โ
โ 32 def main() -> None: โ
โ 33 โ config = parse_args(_CONFIG_CLASS, _CONFIG_YAML) โ
โ โฑ 34 โ train(config) โ
โ 35 โ
โ 36 โ
โ 37 if __name__ == "__main__": โ
โ โ
โ /home/alex.li/git/JupiterCVML/kore/utils/entry_points.py:59 in train โ
โ โ
โ 56 โ # It will be handled manually in the datamodule. โ
โ 57 โ finetune = False if config.finetuning is None else config.finetuning.enable โ
โ 58 โ ckpt_path = None if finetune else config.ckpt_path โ
โ โฑ 59 โ trainer.fit(task, data_module, ckpt_path=ckpt_path) โ
โ 60 โ
โ 61 โ
โ 62 def predict(config: BasePredictionConfig) -> None: โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/pytorch/trainer/traine โ
โ r.py:532 in fit โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/pytorch/trainer/call.p โ
โ y:67 in _call_and_handle_interrupt โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/pytorch/trainer/traine โ
โ r.py:1003 in _teardown โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/pytorch/strategies/ddp โ
โ .py:430 in teardown โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/pytorch/strategies/par โ
โ allel.py:126 in teardown โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/pytorch/strategies/str โ
โ ategy.py:494 in teardown โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/fabric/utilities/optim โ
โ izer.py:28 in _optimizers_to_device โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/fabric/utilities/optim โ
โ izer.py:34 in _optimizer_to_device โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning_utilities/core/apply_f โ
โ unc.py:52 in apply_to_collection โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning_utilities/core/apply_f โ
โ unc.py:104 in _apply_to_collection_slow โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning_utilities/core/apply_f โ
โ unc.py:96 in _apply_to_collection_slow โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/fabric/utilities/apply โ
โ _func.py:101 in move_data_to_device โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning_utilities/core/apply_f โ
โ unc.py:64 in apply_to_collection โ
โ โ
โ /home/alex.li/miniconda3/envs/cvml/lib/python3.10/site-packages/lightning/fabric/utilities/apply โ
โ _func.py:95 in batch_to โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
RuntimeError: CUDA error: device-side assert triggered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Ok, I talked with the supercluster manager, it seems that a GPU was failing somehow in the cluster my job was allocated on.