RuntimeError: CUDA error: device-side assert triggered, LayoutLM Fine-Tuning

JIMUEL_SERVANDIL · July 29, 2023, 6:27am

I am trying to run LayoutLM, with the following arguments:

!  CUDA_LAUNCH_BLOCKING=1 python run_seq_labeling.py \
                            --data_dir "{dataset_dir}" \
                            --labels "{label_file}" \
                            --model_name_or_path "{pretrained_model_folder_input}" \
                            --model_type layoutlm \
                            --max_seq_length 512\
                            --do_lower_case \
                            --do_train \
                            --num_train_epochs 1 \
                            --logging_steps 50 \
                            --save_steps -1 \
                            --output_dir /kaggle/working/output \
                            --overwrite_output_dir \
                            --per_gpu_train_batch_size 8 \
                            --per_gpu_eval_batch_size 16

However wih these arguments, this error occurs:

...
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1603729138878/work/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [157,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Iteration:   2%|▋                               | 3/140 [00:02<01:32,  1.48it/s]
Epoch:   0%|                                              | 0/1 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "run_seq_labeling.py", line 832, in <module>
    main()
  File "run_seq_labeling.py", line 725, in main
    args, train_dataset, model, tokenizer, labels, pad_token_label_id
  File "run_seq_labeling.py", line 240, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 224, in forward
    head_mask=head_mask,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 178, in forward
    input_ids, bbox, position_ids=position_ids, token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 102, in forward
    + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered

When I tried to change the max_seq_length to 4 instead of 512, this error disappears. How can I find out what is the one causing the error and how to fix it?, as the LayoutLM requires the max_seq_length to be 512?

For Reference:

Kaggle Notebook: Here

ptrblck · July 29, 2023, 6:40am

An indexing operation fails so you would either need to run the code on the CPU or with CUDA_LAUNCH_BLOCKING=1 python script.py args to narrow down which line of code and indexing operation fails. Since you are using a language model I would guess an embedding layer might fail and you should thus check your inputs.

JIMUEL_SERVANDIL · July 30, 2023, 1:44pm

I added a print statement where it prints the invalid label indices, what is wrong with these indices?

Invalid labels indices (no truncation): [[0, 0], [0, 2], [0, 3], [0, 4], [0, 6], [0, 7], [0, 8], [0, 10], [0, 16], [0, 18], [0, 22], [0, 24], [0, 25], [0, 26], [0, 28], [0, 29], [0, 30], [0, 31], [0, 32], [0, 33], [0, 34], [0, 35], [0, 36], [0, 40], [0, 42], [0, 45], [0, 46], [0, 50], [0, 51], [0, 55], [0, 58], [0, 59], [0, 60], [0, 61], [0, 62], [0, 63], [0, 64], [0, 65], [0, 66], [0, 68], [0, 69], [0, 70], [0, 71], [0, 73], [0, 75], [0, 76], [0, 77], [0, 78], [0, 80], [0, 81], [0, 82], [0, 85], [0, 87], [0, 88], [0, 89], [0, 90], [0, 91], [0, 92], [0, 93], [0, 94], [0, 96], [0, 98], [0, 99], [0, 101], [0, 102], [0, 104], [0, 105], [0, 106], [0, 107], [0, 108], [0, 109], [0, 111], [0, 112], [0, 113], [0, 114], [0, 115], [0, 116], [0, 117], [0, 118], [0, 119], [0, 121], [0, 122], [0, 123], [0, 124], [0, 125], [0, 126], [0, 127], [0, 129], [0, 130], [0, 131], [0, 133], [0, 134], [0, 136], [0, 139], [0, 140], [0, 142], [0, 143], [0, 146], [0, 147], [0, 150], [0, 151], [0, 153], [0, 155], [0, 156], [0, 158], [0, 161], [0, 162], [0, 164], [0, 165], [0, 167], [0, 168], [0, 170], [0, 171], [0, 172], [0, 173], [0, 175], [0, 176], [0, 179], [0, 181], [0, 185], [0, 195], [0, 196], [0, 199], [0, 200], [0, 203], [0, 205], [0, 206], [0, 207], [0, 208], [0, 209], [0, 210], [0, 212], [0, 214], [0, 217], [0, 218], [0, 219], [0, 220], [0, 221], [0, 222], [0, 223], [0, 224], [0, 225], [0, 226], [0, 227], [0, 228], [0, 229], [0, 231], [0, 232], [0, 233], [0, 235], [0, 236], [0, 237], [0, 238], [0, 241], [0, 243], [0, 244], [0, 245], [0, 246], [0, 247], [0, 248], [0, 249], [0, 250], [0, 251], [0, 252], [0, 253], [0, 254], [0, 255], [0, 256], [0, 257], [0, 258], [0, 259], [0, 275], [0, 276], [0, 280], [0, 281], [0, 283], [0, 287], [0, 293], [0, 296], [0, 299], [0, 304], [0, 306], [0, 310], [0, 311], [0, 313], [0, 314], [0, 315], [0, 316], [0, 317], [0, 318], [0, 319], [0, 320], [0, 321], [0, 322], [0, 330], [0, 331], [0, 332], [0, 333], [0, 334], [0, 335], [0, 336], [0, 337], [0, 338], [0, 339], [0, 340], [0, 341], [0, 342], [0, 343], [0, 344], [0, 345], [0, 346], [0, 347], [0, 348], [0, 349], [0, 350], [0, 351], [0, 352], [0, 353], [0, 354], [0, 355], [0, 356], [0, 357], [0, 358], [0, 359], [0, 360], [0, 361], [0, 362], [0, 363], [0, 364], [0, 365], [0, 366], [0, 367], [0, 368], [0, 369], [0, 370], [0, 371], [0, 372], [0, 373], [0, 374], [0, 375], [0, 376], [0, 377], [0, 378], [0, 379], [0, 380], [0, 381], [0, 382], [0, 383], [0, 384], [0, 385], [0, 386], [0, 387], [0, 388], [0, 389], [0, 390], [0, 391], [0, 392], [0, 393], [0, 394], [0, 395], [0, 396], [0, 397], [0, 398], [0, 399], [0, 400], [0, 401], [0, 402], [0, 403], [0, 404], [0, 405], [0, 406], [0, 407], [0, 408], [0, 409], [0, 410], [0, 411], [0, 412], [0, 413], [0, 414], [0, 415], [0, 416], [0, 417], [0, 418], [0, 419], [0, 420], [0, 421], [0, 422], [0, 423], [0, 424], [0, 425], [0, 426], [0, 427], [0, 428], [0, 429], [0, 430], [0, 431], [0, 432], [0, 433], [0, 434], [0, 435], [0, 436], [0, 437], [0, 438], [0, 439], [0, 440], [0, 441], [0, 442], [0, 443], [0, 444], [0, 445], [0, 446], [0, 447], [0, 448], [0, 449], [0, 450], [0, 451], [0, 452], [0, 453], [0, 454], [0, 455], [0, 456], [0, 457], [0, 458], [0, 459], [0, 460], [0, 461], [0, 462], [0, 463], [0, 464], [0, 465], [0, 466], [0, 467], [0, 468], [0, 469], [0, 470], [0, 471], [0, 472], [0, 473], [0, 474], [0, 475], [0, 476], [0, 477], [0, 478], [0, 479], [0, 480], [0, 481], [0, 482], [0, 483], [0, 484], [0, 485], [0, 486], [0, 487], [0, 488], [0, 489], [0, 490], [0, 491], [0, 492], [0, 493], [0, 494], [0, 495], [0, 496], [0, 497], [0, 498], [0, 499], [0, 500], [0, 501], [0, 502], [0, 503], [0, 504], [0, 505], [0, 506], [0, 507], [0, 508], [0, 509], [0, 510], [0, 511], [1, 0], [1, 2], [1, 3], [1, 5], [1, 12], [1, 15], [1, 17], [1, 18], [1, 19], [1, 21], [1, 22], [1, 23], [1, 25], [1, 26], [1, 27], [1, 28], [1, 29], [1, 30], [1, 31], [1, 32], [1, 33], [1, 34], [1, 35], [1, 37], [1, 38], [1, 41], [1, 42], [1, 46], [1, 49], [1, 50], [1, 51], [1, 52], [1, 53], [1, 55], [1, 56], [1, 58], [1, 59], [1, 60], [1, 61], [1, 63], [1, 65], [1, 66], [1, 67], [1, 68], [1, 69], [1, 71], [1, 72], [1, 73], [1, 76], [1, 78], [1, 79], [1, 80], [1, 81], [1, 82], [1, 83], [1, 84], [1, 85], [1, 90], [1, 92], [1, 93], [1, 94], [1, 95], [1, 96], [1, 97], [1, 99], [1, 100], [1, 101], [1, 102], [1, 103], [1, 104], [1, 105], [1, 106], [1, 108], [1, 109], [1, 110], [1, 111], [1, 112], [1, 113], [1, 114], [1, 116], [1, 117], [1, 118], [1, 121], [1, 122], [1, 124], [1, 125], [1, 128], [1, 129], [1, 132], [1, 133], [1, 135], [1, 137], [1, 138], [1, 140], [1, 141], [1, 142], [1, 144], [1, 145], [1, 149], [1, 150], [1, 152], [1, 153], [1, 154], [1, 155], [1, 157], [1, 158], [1, 161], [1, 163], [1, 165], [1, 167], [1, 177], [1, 178], [1, 181], [1, 182], [1, 185], [1, 187], [1, 188], [1, 189], [1, 190], [1, 191], [1, 192], [1, 194], [1, 196], [1, 199], [1, 200], [1, 201], [1, 202], [1, 203], [1, 204], [1, 205], [1, 206], [1, 207], [1, 208], [1, 209], [1, 210], [1, 211], [1, 213], [1, 214], [1, 215], [1, 217], [1, 218], [1, 219], [1, 220], [1, 222], [1, 223], [1, 224], [1, 225], [1, 226], [1, 229], [1, 231], [1, 232], [1, 233], [1, 234], [1, 235], [1, 236], [1, 237], [1, 238], [1, 239], [1, 240], [1, 241], [1, 242], [1, 243], [1, 244], [1, 245], [1, 246], [1, 262], [1, 263], [1, 267], [1, 268], [1, 270], [1, 274], [1, 280], [1, 283], [1, 286], [1, 291], [1, 293], [1, 297], [1, 298], [1, 300], [1, 301], [1, 302], [1, 303], [1, 304], [1, 305], [1, 306], [1, 307], [1, 308], [1, 309], [1, 316], [1, 317], [1, 318], [1, 319], [1, 320], [1, 321], [1, 322], [1, 323], [1, 324], [1, 325], [1, 326], [1, 327], [1, 328], [1, 329], [1, 330], [1, 331], [1, 332], [1, 333], [1, 334], [1, 335], [1, 336], [1, 337], [1, 338], [1, 339], [1, 340], [1, 341], [1, 342], [1, 343], [1, 344], [1, 345], [1, 346], [1, 347], [1, 348], [1, 349], [1, 350], [1, 351], [1, 352], [1, 353], [1, 354], [1, 355], [1, 356], [1, 357], [1, 358], [1, 359], [1, 360], [1, 361], [1, 362], [1, 363], [1, 364], [1, 365], [1, 366], [1, 367], [1, 368], [1, 369], [1, 370], [1, 371], [1, 372], [1, 373], [1, 374], [1, 375], [1, 376], [1, 377], [1, 378], [1, 379], [1, 380], [1, 381], [1, 382], [1, 383], [1, 384], [1, 385], [1, 386], [1, 387], [1, 388], [1, 389], [1, 390], [1, 391], [1, 392], [1, 393], [1, 394], [1, 395], [1, 396], [1, 397], [1, 398], [1, 399], [1, 400], [1, 401], [1, 402], [1, 403], [1, 404], [1, 405], [1, 406], [1, 407], [1, 408], [1, 409], [1, 410], [1, 411], [1, 412], [1, 413], [1, 414], [1, 415], [1, 416], [1, 417], [1, 418], [1, 419], [1, 420], [1, 421], [1, 422], [1, 423], [1, 424], [1, 425], [1, 426], [1, 427], [1, 428], [1, 429], [1, 430], [1, 431], [1, 432], [1, 433], [1, 434], [1, 435], [1, 436], [1, 437], [1, 438], [1, 439], [1, 440], [1, 441], [1, 442], [1, 443], [1, 444], [1, 445], [1, 446], [1, 447], [1, 448], [1, 449], [1, 450], [1, 451], [1, 452], [1, 453], [1, 454], [1, 455], [1, 456], [1, 457], [1, 458], [1, 459], [1, 460], [1, 461], [1, 462], [1, 463], [1, 464], [1, 465], [1, 466], [1, 467], [1, 468], [1, 469], [1, 470], [1, 471], [1, 472], [1, 473], [1, 474], [1, 475], [1, 476], [1, 477], [1, 478], [1, 479], [1, 480], [1, 481], [1, 482], [1, 483], [1, 484], [1, 485], [1, 486], [1, 487], [1, 488], [1, 489], [1, 490], [1, 491], [1, 492], [1, 493], [1, 494], [1, 495], [1, 496], [1, 497], [1, 498], [1, 499], [1, 500], [1, 501], [1, 502], [1, 503], [1, 504], [1, 505], [1, 506], [1, 507], [1, 508], [1, 509], [1, 510], [1, 511], [2, 0], [2, 2], [2, 3], [2, 4], [2, 6], [2, 13], [2, 16], [2, 18], [2, 19], [2, 20], [2, 22], [2, 23], [2, 24], [2, 26], [2, 27], [2, 28], [2, 29], [2, 30], [2, 31], [2, 32], [2, 33], [2, 34], [2, 35], [2, 36], [2, 38], [2, 39], [2, 42], [2, 43], [2, 47], [2, 50], [2, 51], [2, 52], [2, 53], [2, 54], [2, 56], [2, 57], [2, 59], [2, 60], [2, 61], [2, 62], [2, 64], [2, 66], [2, 67], [2, 68], [2, 69], [2, 71], [2, 73], [2, 74], [2, 75], [2, 77], [2, 78], [2, 79], [2, 80], [2, 81], [2, 82], [2, 83], [2, 84], [2, 86], [2, 87], [2, 88], [2, 91], [2, 93], [2, 94], [2, 95], [2, 96], [2, 97], [2, 98], [2, 99], [2, 100], [2, 104], [2, 105], [2, 109], [2, 110], [2, 111], [2, 112], [2, 113], [2, 114], [2, 115], [2, 116], [2, 118], [2, 119], [2, 123], [2, 124], [2, 125], [2, 128], [2, 129], [2, 131], [2, 132], [2, 136], [2, 137], [2, 140], [2, 141], [2, 143], [2, 145], [2, 146], [2, 148], [2, 151], [2, 152], [2, 156], [2, 157], [2, 159], [2, 160], [2, 161], [2, 162], [2, 164], [2, 165], [2, 168], [2, 170], [2, 172], [2, 174], [2, 184], [2, 185], [2, 188], [2, 189], [2, 192], [2, 194], [2, 195], [2, 196], [2, 197], [2, 198], [2, 199], [2, 201], [2, 203], [2, 206], [2, 207], [2, 208], [2, 209], [2, 210], [2, 211], [2, 212], [2, 213], [2, 214], [2, 215], [2, 216], [2, 217], [2, 218], [2, 220], [2, 221], [2, 222], [2, 224], [2, 225], [2, 226], [2, 227], [2, 229], [2, 230], [2, 231], [2, 232], [2, 233], [2, 236], [2, 238], [2, 239], [2, 240], [2, 241], [2, 242], [2, 243], [2, 244], [2, 245], [2, 246], [2, 247], [2, 248], [2, 249], [2, 250], [2, 251], [2, 252], [2, 253], [2, 269], [2, 270], [2, 274], [2, 275], [2, 277], [2, 281], [2, 287], [2, 290], [2, 293], [2, 298], [2, 300], [2, 304], [2, 305], [2, 307], [2, 308], [2, 309], [2, 310], [2, 311], [2, 312], [2, 313], [2, 314], [2, 315], [2, 316], [2, 323], [2, 324], [2, 325], [2, 326], [2, 327], [2, 328], [2, 329], [2, 330], [2, 331], [2, 332], [2, 333], [2, 334], [2, 335], [2, 336], [2, 337], [2, 338], [2, 339], [2, 340], [2, 341], [2, 342], [2, 343], [2, 344], [2, 345], [2, 346], [2, 347], [2, 348], [2, 349], [2, 350], [2, 351], [2, 352], [2, 353], [2, 354], [2, 355], [2, 356], [2, 357], [2, 358], [2, 359], [2, 360], [2, 361], [2, 362], [2, 363], [2, 364], [2, 365], [2, 366], [2, 367], [2, 368], [2, 369], [2, 370], [2, 371], [2, 372], [2, 373], [2, 374], [2, 375], [2, 376], [2, 377], [2, 378], [2, 379], [2, 380], [2, 381], [2, 382], [2, 383], [2, 384], [2, 385], [2, 386], [2, 387], [2, 388], [2, 389], [2, 390], [2, 391], [2, 392], [2, 393], [2, 394], [2, 395], [2, 396], [2, 397], [2, 398], [2, 399], [2, 400], [2, 401], [2, 402], [2, 403], [2, 404], [2, 405], [2, 406], [2, 407], [2, 408], [2, 409], [2, 410], [2, 411], [2, 412], [2, 413], [2, 414], [2, 415], [2, 416], [2, 417], [2, 418], [2, 419], [2, 420], [2, 421], [2, 422], [2, 423], [2, 424], [2, 425], [2, 426], [2, 427], [2, 428], [2, 429], [2, 430], [2, 431], [2, 432], [2, 433], [2, 434], [2, 435], [2, 436], [2, 437], [2, 438], [2, 439], [2, 440], [2, 441], [2, 442], [2, 443], [2, 444], [2, 445], [2, 446], [2, 447], [2, 448], [2, 449], [2, 450], [2, 451], [2, 452], [2, 453], [2, 454], [2, 455], [2, 456], [2, 457], [2, 458], [2, 459], [2, 460], [2, 461], [2, 462], [2, 463], [2, 464], [2, 465], [2, 466], [2, 467], [2, 468], [2, 469], [2, 470], [2, 471], [2, 472], [2, 473], [2, 474], [2, 475], [2, 476], [2, 477], [2, 478], [2, 479], [2, 480], [2, 481], [2, 482], [2, 483], [2, 484], [2, 485], [2, 486], [2, 487], [2, 488], [2, 489], [2, 490], [2, 491], [2, 492], [2, 493], [2, 494], [2, 495], [2, 496], [2, 497], [2, 498], [2, 499], [2, 500], [2, 501], [2, 502], [2, 503], [2, 504], [2, 505], [2, 506], [2, 507], [2, 508], [2, 509], [2, 510], [2, 511], [3, 0], [3, 2], [3, 3], [3, 4], [3, 6], [3, 7], [3, 8], [3, 10], [3, 16], [3, 18], [3, 22], [3, 24], [3, 25], [3, 26], [3, 28], [3, 29], [3, 30], [3, 31], [3, 32], [3, 33], [3, 34], [3, 35], [3, 36], [3, 40], [3, 42], [3, 45], [3, 46], [3, 49], [3, 51], [3, 52], [3, 56], [3, 59], [3, 60], [3, 61], [3, 62], [3, 63], [3, 64], [3, 65], [3, 66], [3, 68], [3, 69], [3, 70], [3, 71], [3, 73], [3, 75], [3, 76], [3, 77], [3, 79], [3, 80], [3, 81], [3, 82], [3, 83], [3, 84], [3, 86], [3, 87], [3, 88], [3, 90], [3, 91], [3, 92], [3, 93], [3, 95], [3, 96], [3, 97], [3, 100], [3, 102], [3, 103], [3, 104], [3, 105], [3, 106], [3, 107], [3, 108], [3, 109], [3, 113], [3, 114], [3, 116], [3, 117], [3, 119], [3, 120], [3, 121], [3, 122], [3, 123], [3, 124], [3, 125], [3, 126], [3, 128], [3, 130], [3, 131], [3, 133], [3, 134], [3, 138], [3, 139], [3, 140], [3, 141], [3, 142], [3, 143], [3, 144], [3, 146], [3, 147], [3, 148], [3, 150], [3, 151], [3, 152], [3, 153], [3, 154], [3, 155], [3, 156], [3, 158], [3, 159], [3, 160], [3, 164], [3, 165], [3, 167], [3, 168], [3, 169], [3, 172], [3, 173], [3, 176], [3, 177], [3, 179], [3, 181], [3, 182], [3, 183], [3, 185], [3, 186], [3, 187], [3, 189], [3, 190], [3, 194], [3, 195], [3, 197], [3, 198],

…

[12, 233], [12, 234], [12, 235], [12, 236], [12, 237], [12, 238], [12, 240], [12, 241], [12, 242], [12, 244], [12, 245], [12, 246], [12, 247], [12, 250], [12, 252], [12, 253], [12, 254], [12, 255], [12, 256], [12, 257], [12, 258], [12, 259], [12, 260], [12, 261], [12, 262], [12, 263], [12, 264], [12, 265], [12, 266], [12, 267], [12, 283], [12, 284], [12, 288], [12, 289], [12, 291], [12, 295], [12, 301], [12, 304], [12, 307], [12, 312], [12, 314], [12, 318], [12, 319], [12, 321], [12, 322], [12, 323], [12, 324], [12, 325], [12, 326], [12, 327], [12, 328], [12, 329], [12, 330], [12, 331], [12, 332], [12, 333], [12, 334], [12, 335], [12, 336], [12, 337], [12, 338], [12, 339], [12, 340], [12, 341], [12, 342], [12, 343], [12, 344], [12, 345], [12, 346], [12, 347], [12, 348], [12, 349], [12, 350], [12, 351], [12, 352], [12, 353], [12, 354], [12, 355], [12, 356], [12, 357], [12, 358], [12, 359], [12, 360], [12, 361], [12, 362], [12, 363], [12, 364], [12, 365], [12, 366], [12, 367], [12, 368], [12, 369], [12, 370], [12, 371], [12, 372], [12, 373], [12, 374], [12, 375], [12, 376], [12, 377], [12, 378], [12, 379], [12, 380], [12, 381], [12, 382], [12, 383], [12, 384], [12, 385], [12, 386], [12, 387], [12, 388], [12, 389], [12, 390], [12, 391], [12, 392], [12, 393], [12, 394], [12, 395], [12, 396], [12, 397], [12, 398], [12, 399], [12, 400], [12, 401], [12, 402], [12, 403], [12, 404], [12, 405], [12, 406], [12, 407], [12, 408], [12, 409], [12, 410], [12, 411], [12, 412], [12, 413], [12, 414], [12, 415], [12, 416], [12, 417], [12, 418], [12, 419], [12, 420], [12, 421], [12, 422], [12, 423], [12, 424], [12, 425], [12, 426], [12, 427], [12, 428], [12, 429], [12, 430], [12, 431], [12, 432], [12, 433], [12, 434], [12, 435], [12, 436], [12, 437], [12, 438], [12, 439], [12, 440], [12, 441], [12, 442], [12, 443], [12, 444], [12, 445], [12, 446], [12, 447], [12, 448], [12, 449], [12, 450], [12, 451], [12, 452], [12, 453], [12, 454], [12, 455], [12, 456], [12, 457], [12, 458], [12, 459], [12, 460], [12, 461], [12, 462], [12, 463], [12, 464], [12, 465], [12, 466], [12, 467], [12, 468], [12, 469], [12, 470], [12, 471], [12, 472], [12, 473], [12, 474], [12, 475], [12, 476], [12, 477], [12, 478], [12, 479], [12, 480], [12, 481], [12, 482], [12, 483], [12, 484], [12, 485], [12, 486], [12, 487], [12, 488], [12, 489], [12, 490], [12, 491], [12, 492], [12, 493], [12, 494], [12, 495], [12, 496], [12, 497], [12, 498], [12, 499], [12, 500], [12, 501], [12, 502], [12, 503], [12, 504], [12, 505], [12, 506], [12, 507], [12, 508], [12, 509], [12, 510], [12, 511], [13, 0], [13, 2], [13, 3], [13, 4], [13, 6], [13, 7], [13, 8], [13, 10], [13, 16], [13, 18], [13, 22], [13, 24], [13, 25], [13, 26], [13, 28], [13, 29], [13, 30], [13, 31], [13, 32], [13, 33], [13, 34], [13, 35], [13, 36], [13, 40], [13, 42], [13, 45], [13, 46], [13, 50], [13, 51], [13, 55], [13, 58], [13, 59], [13, 60], [13, 61], [13, 62], [13, 64], [13, 65], [13, 66], [13, 68], [13, 69], [13, 70], [13, 71], [13, 73], [13, 75], [13, 76], [13, 77], [13, 78], [13, 80], [13, 81], [13, 82], [13, 83], [13, 84], [13, 85], [13, 87], [13, 88], [13, 89], [13, 91], [13, 92], [13, 93], [13, 94], [13, 95], [13, 97], [13, 98], [13, 99], [13, 102], [13, 104], [13, 105], [13, 106], [13, 107], [13, 108], [13, 109], [13, 110], [13, 111], [13, 113], [13, 115], [13, 116], [13, 118], [13, 119], [13, 121], [13, 122], [13, 123], [13, 124], [13, 125], [13, 126], [13, 127], [13, 128], [13, 130], [13, 131], [13, 132], [13, 135], [13, 136], [13, 138], [13, 139], [13, 142], [13, 143], [13, 146], [13, 147], [13, 149], [13, 151], [13, 152], [13, 154], [13, 157], [13, 158], [13, 162], [13, 163], [13, 165], [13, 166], [13, 167], [13, 168], [13, 170], [13, 171], [13, 174], [13, 178], [13, 180], [13, 190], [13, 191], [13, 194], [13, 195], [13, 198], [13, 200], [13, 201], [13, 202], [13, 203], [13, 204], [13, 205], [13, 207], [13, 209], [13, 212], [13, 213], [13, 214], [13, 215], [13, 216], [13, 217], [13, 218], [13, 219], [13, 220], [13, 221], [13, 222], [13, 223], [13, 224], [13, 226], [13, 227], [13, 228], [13, 230], [13, 231], [13, 232], [13, 233], [13, 236], [13, 238], [13, 239], [13, 240], [13, 241], [13, 242], [13, 243], [13, 244], [13, 245], [13, 246], [13, 247], [13, 248], [13, 249], [13, 250], [13, 251], [13, 252], [13, 253], [13, 254], [13, 270], [13, 271], [13, 275], [13, 276], [13, 278], [13, 282], [13, 288], [13, 291], [13, 294], [13, 299], [13, 301], [13, 305], [13, 306], [13, 308], [13, 309], [13, 310], [13, 311], [13, 312], [13, 313], [13, 314], [13, 315], [13, 316], [13, 317], [13, 324], [13, 325], [13, 326], [13, 327], [13, 328], [13, 329], [13, 330], [13, 331], [13, 332], [13, 333], [13, 334], [13, 335], [13, 336], [13, 337], [13, 338], [13, 339], [13, 340], [13, 341], [13, 342], [13, 343], [13, 344], [13, 345], [13, 346], [13, 347], [13, 348], [13, 349], [13, 350], [13, 351], [13, 352], [13, 353], [13, 354], [13, 355], [13, 356], [13, 357], [13, 358], [13, 359], [13, 360], [13, 361], [13, 362], [13, 363], [13, 364], [13, 365], [13, 366], [13, 367], [13, 368], [13, 369], [13, 370], [13, 371], [13, 372], [13, 373], [13, 374], [13, 375], [13, 376], [13, 377], [13, 378], [13, 379], [13, 380], [13, 381], [13, 382], [13, 383], [13, 384], [13, 385], [13, 386], [13, 387], [13, 388], [13, 389], [13, 390], [13, 391], [13, 392], [13, 393], [13, 394], [13, 395], [13, 396], [13, 397], [13, 398], [13, 399], [13, 400], [13, 401], [13, 402], [13, 403], [13, 404], [13, 405], [13, 406], [13, 407], [13, 408], [13, 409], [13, 410], [13, 411], [13, 412], [13, 413], [13, 414], [13, 415], [13, 416], [13, 417], [13, 418], [13, 419], [13, 420], [13, 421], [13, 422], [13, 423], [13, 424], [13, 425], [13, 426], [13, 427], [13, 428], [13, 429], [13, 430], [13, 431], [13, 432], [13, 433], [13, 434], [13, 435], [13, 436], [13, 437], [13, 438], [13, 439], [13, 440], [13, 441], [13, 442], [13, 443], [13, 444], [13, 445], [13, 446], [13, 447], [13, 448], [13, 449], [13, 450], [13, 451], [13, 452], [13, 453], [13, 454], [13, 455], [13, 456], [13, 457], [13, 458], [13, 459], [13, 460], [13, 461], [13, 462], [13, 463], [13, 464], [13, 465], [13, 466], [13, 467], [13, 468], [13, 469], [13, 470], [13, 471], [13, 472], [13, 473], [13, 474], [13, 475], [13, 476], [13, 477], [13, 478], [13, 479], [13, 480], [13, 481], [13, 482], [13, 483], [13, 484], [13, 485], [13, 486], [13, 487], [13, 488], [13, 489], [13, 490], [13, 491], [13, 492], [13, 493], [13, 494], [13, 495], [13, 496], [13, 497], [13, 498], [13, 499], [13, 500], [13, 501], [13, 502], [13, 503], [13, 504], [13, 505], [13, 506], [13, 507], [13, 508], [13, 509], [13, 510], [13, 511], [14, 0], [14, 2], [14, 3], [14, 5], [14, 12], [14, 15], [14, 17], [14, 18], [14, 19], [14, 21], [14, 22], [14, 23], [14, 25], [14, 26], [14, 27], [14, 28], [14, 29], [14, 30], [14, 31], [14, 32], [14, 33], [14, 34], [14, 35], [14, 37], [14, 38], [14, 41], [14, 42], [14, 46], [14, 49], [14, 50], [14, 51], [14, 52], [14, 53], [14, 55], [14, 56], [14, 58], [14, 59], [14, 60], [14, 61], [14, 63], [14, 65], [14, 66], [14, 67], [14, 68], [14, 70], [14, 71], [14, 72], [14, 75], [14, 77], [14, 78], [14, 79], [14, 80], [14, 81], [14, 82], [14, 83], [14, 84], [14, 86], [14, 87], [14, 88], [14, 89], [14, 93], [14, 94], [14, 95], [14, 96], [14, 97], [14, 98], [14, 100], [14, 101], [14, 102], [14, 103], [14, 104], [14, 105], [14, 106], [14, 107], [14, 109], [14, 110], [14, 111], [14, 112], [14, 113], [14, 114], [14, 116], [14, 117], [14, 118], [14, 119], [14, 120], [14, 122], [14, 123], [14, 126], [14, 127], [14, 129], [14, 130], [14, 133], [14, 134], [14, 137], [14, 138], [14, 140], [14, 142], [14, 143], [14, 145], [14, 148], [14, 149], [14, 151], [14, 152], [14, 154], [14, 155], [14, 157], [14, 159], [14, 161], [14, 162], [14, 165], [14, 168], [14, 170], [14, 172], [14, 182], [14, 183], [14, 186], [14, 187], [14, 190], [14, 192], [14, 193], [14, 194], [14, 195], [14, 196], [14, 197], [14, 199], [14, 201], [14, 204], [14, 205], [14, 206], [14, 207], [14, 208], [14, 209], [14, 210], [14, 211], [14, 212], [14, 213], [14, 214], [14, 215], [14, 216], [14, 218], [14, 219], [14, 220], [14, 222], [14, 223], [14, 224], [14, 225], [14, 227], [14, 228], [14, 229], [14, 230], [14, 231], [14, 234], [14, 236], [14, 237], [14, 238], [14, 239], [14, 240], [14, 241], [14, 242], [14, 243], [14, 244], [14, 245], [14, 246], [14, 247], [14, 248], [14, 249], [14, 250], [14, 251], [14, 267], [14, 268], [14, 272], [14, 273], [14, 275], [14, 279], [14, 285], [14, 290], [14, 295], [14, 297], [14, 301], [14, 302], [14, 304], [14, 305], [14, 306], [14, 307], [14, 308], [14, 309], [14, 310], [14, 311], [14, 312], [14, 313], [14, 320], [14, 321], [14, 322], [14, 323], [14, 324], [14, 325], [14, 326], [14, 327], [14, 328], [14, 329], [14, 330], [14, 331], [14, 332], [14, 333], [14, 334], [14, 335], [14, 336], [14, 337], [14, 338], [14, 339], [14, 340], [14, 341], [14, 342], [14, 343], [14, 344], [14, 345], [14, 346], [14, 347], [14, 348], [14, 349], [14, 350], [14, 351], [14, 352], [14, 353], [14, 354], [14, 355], [14, 356], [14, 357], [14, 358], [14, 359], [14, 360], [14, 361], [14, 362], [14, 363], [14, 364], [14, 365], [14, 366], [14, 367], [14, 368], [14, 369], [14, 370], [14, 371], [14, 372], [14, 373], [14, 374], [14, 375], [14, 376], [14, 377], [14, 378], [14, 379], [14, 380], [14, 381], [14, 382], [14, 383], [14, 384], [14, 385], [14, 386], [14, 387], [14, 388], [14, 389], [14, 390], [14, 391], [14, 392], [14, 393], [14, 394], [14, 395], [14, 396], [14, 397], [14, 398], [14, 399], [14, 400], [14, 401], [14, 402], [14, 403], [14, 404], [14, 405], [14, 406], [14, 407], [14, 408], [14, 409], [14, 410], [14, 411], [14, 412], [14, 413], [14, 414], [14, 415], [14, 416], [14, 417], [14, 418], [14, 419], [14, 420], [14, 421], [14, 422], [14, 423], [14, 424], [14, 425], [14, 426], [14, 427], [14, 428], [14, 429], [14, 430], [14, 431], [14, 432], [14, 433], [14, 434], [14, 435], [14, 436], [14, 437], [14, 438], [14, 439], [14, 440], [14, 441], [14, 442], [14, 443], [14, 444], [14, 445], [14, 446], [14, 447], [14, 448], [14, 449], [14, 450], [14, 451], [14, 452], [14, 453], [14, 454], [14, 455], [14, 456], [14, 457], [14, 458], [14, 459], [14, 460], [14, 461], [14, 462], [14, 463], [14, 464], [14, 465], [14, 466], [14, 467], [14, 468], [14, 469], [14, 470], [14, 471], [14, 472], [14, 473], [14, 474], [14, 475], [14, 476], [14, 477], [14, 478], [14, 479], [14, 480], [14, 481], [14, 482], [14, 483], [14, 484], [14, 485], [14, 486], [14, 487], [14, 488], [14, 489], [14, 490], [14, 491], [14, 492], [14, 493], [14, 494], [14, 495], [14, 496], [14, 497], [14, 498], [14, 499], [14, 500], [14, 501], [14, 502], [14, 503], [14, 504], [14, 505], [14, 506], [14, 507], [14, 508], [14, 509], [14, 510], [14, 511], [15, 0], [15, 2], [15, 3], [15, 5], [15, 10], [15, 13], [15, 16], [15, 18], [15, 19], [15, 20], [15, 22], [15, 23], [15, 24], [15, 26], [15, 27], [15, 28], [15, 29], [15, 30], [15, 31], [15, 32], [15, 33], [15, 34], [15, 35], [15, 36], [15, 38], [15, 39], [15, 42], [15, 43], [15, 47], [15, 50], [15, 51], [15, 52], [15, 53], [15, 54], [15, 56], [15, 57], [15, 59], [15, 60], [15, 61], [15, 62], [15, 64], [15, 66], [15, 67], [15, 68], [15, 69], [15, 70], [15, 72], [15, 73], [15, 74], [15, 75], [15, 76], [15, 77], [15, 79], [15, 80], [15, 81], [15, 82], [15, 83], [15, 84], [15, 85], [15, 86], [15, 88], [15, 89], [15, 90], [15, 93], [15, 95], [15, 96], [15, 97], [15, 98], [15, 99], [15, 100], [15, 101], [15, 102], [15, 104], [15, 105], [15, 107], [15, 109], [15, 110], [15, 111], [15, 112], [15, 113], [15, 114], [15, 116], [15, 117], [15, 121], [15, 122], [15, 123], [15, 125], [15, 126], [15, 127], [15, 128], [15, 129], [15, 130], [15, 131], [15, 132], [15, 133], [15, 135], [15, 136], [15, 140], [15, 141], [15, 143], [15, 144], [15, 147], [15, 148], [15, 151], [15, 152], [15, 153], [15, 155], [15, 157], [15, 158], [15, 160], [15, 161], [15, 162], [15, 164], [15, 165], [15, 167], [15, 168], [15, 170], [15, 171], [15, 173], [15, 174], [15, 175], [15, 176], [15, 178], [15, 179], [15, 181], [15, 182], [15, 183], [15, 186], [15, 188], [15, 189], [15, 190], [15, 191], [15, 192], [15, 193], [15, 194], [15, 198], [15, 209], [15, 210], [15, 213], [15, 214], [15, 217], [15, 219], [15, 220], [15, 221], [15, 222], [15, 223], [15, 224], [15, 226], [15, 228], [15, 231], [15, 232], [15, 233], [15, 234], [15, 235], [15, 236], [15, 237], [15, 238], [15, 239], [15, 240], [15, 241], [15, 242], [15, 243], [15, 245], [15, 246], [15, 247], [15, 249], [15, 250], [15, 251], [15, 252], [15, 254], [15, 255], [15, 256], [15, 257], [15, 258], [15, 262], [15, 264], [15, 265], [15, 266], [15, 267], [15, 268], [15, 269], [15, 270], [15, 271], [15, 272], [15, 273], [15, 274], [15, 275], [15, 276], [15, 277], [15, 278], [15, 279], [15, 295], [15, 296], [15, 300], [15, 301], [15, 303], [15, 307], [15, 313], [15, 316], [15, 319], [15, 324], [15, 326], [15, 330], [15, 331], [15, 333], [15, 334], [15, 335], [15, 336], [15, 337], [15, 338], [15, 339], [15, 340], [15, 341], [15, 342], [15, 343], [15, 344], [15, 345], [15, 346], [15, 347], [15, 348], [15, 349], [15, 350], [15, 351], [15, 352], [15, 353], [15, 354], [15, 355], [15, 356], [15, 357], [15, 358], [15, 359], [15, 360], [15, 361], [15, 362], [15, 363], [15, 364], [15, 365], [15, 366], [15, 367], [15, 368], [15, 369], [15, 370], [15, 371], [15, 372], [15, 373], [15, 374], [15, 375], [15, 376], [15, 377], [15, 378], [15, 379], [15, 380], [15, 381], [15, 382], [15, 383], [15, 384], [15, 385], [15, 386], [15, 387], [15, 388], [15, 389], [15, 390], [15, 391], [15, 392], [15, 393], [15, 394], [15, 395], [15, 396], [15, 397], [15, 398], [15, 399], [15, 400], [15, 401], [15, 402], [15, 403], [15, 404], [15, 405], [15, 406], [15, 407], [15, 408], [15, 409], [15, 410], [15, 411], [15, 412], [15, 413], [15, 414], [15, 415], [15, 416], [15, 417], [15, 418], [15, 419], [15, 420], [15, 421], [15, 422], [15, 423], [15, 424], [15, 425], [15, 426], [15, 427], [15, 428], [15, 429], [15, 430], [15, 431], [15, 432], [15, 433], [15, 434], [15, 435], [15, 436], [15, 437], [15, 438], [15, 439], [15, 440], [15, 441], [15, 442], [15, 443], [15, 444], [15, 445], [15, 446], [15, 447], [15, 448], [15, 449], [15, 450], [15, 451], [15, 452], [15, 453], [15, 454], [15, 455], [15, 456], [15, 457], [15, 458], [15, 459], [15, 460], [15, 461], [15, 462], [15, 463], [15, 464], [15, 465], [15, 466], [15, 467], [15, 468], [15, 469], [15, 470], [15, 471], [15, 472], [15, 473], [15, 474], [15, 475], [15, 476], [15, 477], [15, 478], [15, 479], [15, 480], [15, 481], [15, 482], [15, 483], [15, 484], [15, 485], [15, 486], [15, 487], [15, 488], [15, 489], [15, 490], [15, 491], [15, 492], [15, 493], [15, 494], [15, 495], [15, 496], [15, 497], [15, 498], [15, 499], [15, 500], [15, 501], [15, 502], [15, 503], [15, 504], [15, 505], [15, 506], [15, 507], [15, 508], [15, 509], [15, 510], [15, 511]]

Position embeddings size: torch.Size([1024, 768])
Maximum index value: tensor(29656, device='cuda:0')
Iteration:   0%|                                          | 0/5 [00:00<?, ?it/s]
Epoch:   0%|                                              | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_seq_labeling.py", line 870, in <module>
    main()
  File "run_seq_labeling.py", line 763, in main
    args, train_dataset, model, tokenizer, labels, pad_token_label_id
  File "run_seq_labeling.py", line 278, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 224, in forward
    head_mask=head_mask,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 178, in forward
    input_ids, bbox, position_ids=position_ids, token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 105, in forward
    embeddings = self.dropout(embeddings)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/dropout.py", line 58, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 983, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: CUDA error: device-side assert triggered

ptrblck · July 30, 2023, 2:28pm

I don’t know as I’m not familiar with your model and its requirements, but did you try to narrow down the failing operation first as suggested?

JIMUEL_SERVANDIL · July 31, 2023, 4:19am

When I tried to run it on CPU, this is the error:

Position embeddings size: torch.Size([1024, 768])
Maximum index value: tensor(29656)
Iteration:   0%|                                          | 0/5 [00:00<?, ?it/s]
Epoch:   0%|                                              | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_seq_labeling.py", line 870, in <module>
    main()
  File "run_seq_labeling.py", line 763, in main
    args, train_dataset, model, tokenizer, labels, pad_token_label_id
  File "run_seq_labeling.py", line 278, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 224, in forward
    head_mask=head_mask,
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 178, in forward
    input_ids, bbox, position_ids=position_ids, token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/layoutlm/modeling/layoutlm.py", line 89, in forward
    bbox[:, :, 2] - bbox[:, :, 0]
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

And when I try to include the CUDA_LAUNCH_BLOCKING=1, the
RuntimeError: CUDA error: device-side assert triggered is being returned

ptrblck · July 31, 2023, 5:51am

Thanks for confirming my guess. Since the embedding layer is indeed failing make sure the inputs contain índices in the range [0, num_embeddings-1].

JIMUEL_SERVANDIL · July 31, 2023, 9:21am

I have added a print statement to print the labels tensor, and I think this is the problematic tensors. What might be the cause of this and how can I fix it?

Epoch:   0%|                                              | 0/1 [00:00<?, ?it/s]
Iteration:   0%|                                          | 0/5 [00:00<?, ?it/s]
Number of Output Units: 9

Num of Labels: 9

Labels Tensor: tensor([[-100,    0, -100, -100, -100,    8, -100, -100, -100,    8, -100,    8,
            8,    8,    8,    8, -100,    8, -100,    8,    8,    8, -100,    8,
         -100, -100, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100,
         -100,    1,    1,    1, -100,    1, -100,    1,    1, -100, -100,    1,
            1,    1, -100, -100,    1,    8,    8, -100,    8,    2, -100, -100,
         -100, -100, -100, -100, -100, -100, -100,    3, -100, -100, -100, -100,
            8, -100,    8, -100, -100, -100, -100,    8, -100, -100, -100,    8,
            8, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100,    8,
         -100,    8, -100, -100,    8, -100, -100,    8, -100, -100, -100, -100,
         -100, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100, -100,
            4, -100, -100, -100, -100, -100, -100, -100,    5, -100, -100, -100,
            8, -100, -100,    8, -100,    8,    8, -100, -100,    6, -100, -100,
            8,    6, -100, -100,    8,    6, -100, -100,    8, -100,    8, -100,
         -100,    8, -100,    8,    7, -100, -100,    8, -100, -100,    6, -100,
         -100,    8, -100, -100, -100, -100,    6, -100, -100,    8,    8, -100,
            8, -100,    8,    8,    8, -100,    8,    8,    8,    8,    8,    8,
            8,    8,    8, -100, -100,    8,    8, -100, -100,    8,    8, -100,
            8, -100, -100, -100, -100, -100, -100,    8, -100,    8, -100,    8,
            8, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100,    8, -100, -100, -100,    8, -100, -100, -100, -100,    8,
            8, -100,    8, -100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100,    8,    8,    8,    8,
            8,    8,    8,    8,    6,    8,    8,    8,    8,    8,    8, -100,
         -100,    8,    8,    8, -100, -100,    8, -100,    8,    8,    8, -100,
            8,    8,    8,    8,    8, -100,    8,    8, -100,    8,    8, -100,
            8,    8,    8,    8, -100,    8, -100,    8,    8,    8, -100, -100,
            8, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,    8,
            8,    8,    8,    8,    8,    8, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100,

ptrblck · July 31, 2023, 11:38am

I don’t know if you are passing the label tensor to the embedding layer, as it would be uncommon, but if so then -100 is out of bounds given the previously mentioned range of [0, num_embeddings-1]. Did you check where the embedding layer is used, which tensor is passed to it, and what its min and max values are?

JIMUEL_SERVANDIL · August 2, 2023, 8:43am

In the layoutlm.py script, the embedding layer is used in the LayoutlmEmbeddings class and there are no labels in the embedding layer. The input tensors were:

input_ids, bbox, token_type_ids, position_ids, input_embeds.

The min max of the values were:

Input IDs: (0, 29656)
BBox: (0, 1000)
Token Type IDs: (0, 0)
Position IDs: (0, 511)
# The input_embeds returns `None`

ptrblck · August 2, 2023, 11:19am

And what is the num_embedding value of the layer since the inputs must contain índices using it as their bound as described before?

jbkamp · September 10, 2024, 2:11pm

I had the same error for BertForTokenClassification. No problem on cpu and mps, but CUDA raised errors. Spent way too much time trying to solve. The error is probably due to the padding class labels (-100) and some mismatch with the tokenizer. I solved it by adding this line after defining the model and the tokenizer, thanks to pytorch - RuntimeError: CUDA error: device-side assert triggered - BART model - Stack Overflow and https://discuss.pytorch.org/t/runtimeerror-cuda-error-device-side-assert-triggered-when-fine-tune-bert/99170/7:

model.resize_token_embeddings(len(tokenizer))