CUDA error: CUBLAS_STATUS_NOT_INITIALIZED

nlp
1

Hi everyone,
I am doing POS tagging, and when computing the backward pass I am getting the following error:
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle).
I tried a bunch of different solutions on the web and nothing worked.
Here is the code that is triggering the error (note that I am using bert-base for classification) :

nl_databatch = data_dict['tr_batch']
        input_ids = nl_databatch['input_ids']
        attention_mask = nl_databatch['attention_mask']
        n_labels = nl_databatch['n_labels']
        input_ids = input_ids.to(device)
        attention_mask = attention_mask.to(device)
        n_labels = n_labels.to(device)
        outputs = model(input_ids, attention_mask)
        outputs = outputs.view(-1, outputs.shape[-1])
        n_labels = n_labels.view(-1)
        
        loss = loss_fn(outputs, n_labels)
        

        loss.backward()

        return loss.item()

It would be a huge help if anyone can help me debug this error, I have a deadline very soon and need to solve this asap.

2

Hello, friend.
I don’t know if it will help, but when I refer to the github below, I think it’s because of the length problem of the tokenizer. (When I saw the case of bert)

Sorry if you’ve already been looking for it.