RuntimeError: "element 0 of tensors does not require grad and does not have a grad_fn"

nlp
1

Hello PyTorch community,

I hope this message finds you well. I am facing an issue while training a comment classification model using PyTorch Lightning with a pre-trained BERT model.

I encountered the following error during the training process:

javascriptCopy code

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

To provide some context, I have already enabled gradients for all parameters of the model using the function enable_gradients(model). However, the error still persists.

The model I am using is based on the aubmindlab/bert-base-arabertv02-twitter pre-trained model, and I noticed that some weights of the BERT model were not initialized properly upon loading. I have ensured that I am using the latest versions of PyTorch, Transformers, and PyTorch Lightning.

I attempted to pretrain the BERT model on a downstream task before training my specific model, but the error remains unresolved.

I am reaching out to the community for any guidance or insights on how to resolve this issue. I would be more than happy to provide additional details about my training environment, code, and model setup if needed.

Thank you in advance for your support and advice. Your expertise and suggestions are highly appreciated.

Best regards,
Hamza

2

I’m unsure but your issue might also be related to this one where it seems that HF models are either detaching the computation graph or are disabling gradient calculations. I don’t see any updates, so unsure if these issues are already forwarded or fixed.
To debug the issue you could check the .grad_fn of intermediate activations and make sure a valid backward function is returned. If the .grad_fn is set to None the tensor is not attached to a computation graph.

3

this is my code :

from pytorch_lightning import Trainer

def enable_gradients(model):
for param in model.parameters():
param.requires_grad = True

datamodule

ucc_data_module = UCC_Data_Module(train_path, val_path, test_path, attributes=attributes, batch_size=config[‘batch_size’])
ucc_data_module.setup()

model

model = UCC_Comment_Classifier()

enable_gradients(model)

trainer and fit

Instantiation of the Lightning Trainer

trainer = Trainer(max_epochs=config[‘n_epochs’], accelerator=‘gpu’, num_sanity_val_steps=1)

try:
trainer.fit(model, ucc_data_module)
torch.save(model.state_dict(), PATH)
except RuntimeError as e:
print(e)