First the error information is here:
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/run.py", line 18, in <module>
main(prog="allennlp")
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/commands/__init__.py", line 70, in main
args.func(args)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/commands/train.py", line 101, in train_model_from_args
args.recover)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/commands/train.py", line 131, in train_model_from_file
return train_model(params, serialization_dir, file_friendly_logging, recover)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/commands/train.py", line 321, in train_model
metrics = trainer.train()
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/training/trainer.py", line 749, in train
train_metrics = self._train_epoch(epoch)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/allennlp/training/trainer.py", line 493, in _train_epoch
loss.backward()
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/jiaofangkai/anaconda3/envs/allennlp-env/lib/python3.7/site-packages/torch/autograd/__init__.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: MAGMA getrf : U(4,4) is 0, U is singular at /opt/conda/conda-bld/pytorch_1532584813488/work/aten/src/THC/generic/THCTensorMathMagma.cu:413
So I calculate the softmax of head_num vectors and get a batch of matrix of shape (batch_size, head_num, encoding_dim) which we called it A. And then I calculate as
S = torch.bmm(A, A.transpose(2,1)) - torch.eye(head_num).cuda()
At last, I calculate the determinant for S[i] and add to the final loss:
self_attention_loss = torch.tensor(0, dtype=torch.float).cuda()
for i in range(batch_size):
self_attention_loss += torch.det(loss_matrix[i])
loss += (self_attention_loss * self._loss_rate / torch.tensor(batch_size, dtype=torch.float).cuda())
It seems that the test codes as below is ok:
import torch
a = torch.Tensor([[0,0],[0,0]])
a.requires_grad=True
b = torch.det(a)
print(b)
b.backward()
print(a.grad)
The output is:
tensor(0., grad_fn=<DetBackward>)
tensor([[0., 0.],
[0., 0.]])
So I don’t know where the reason is? Is it because the determinant is zero?
Thanks you all a lot!