RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/

1

I have two GPUs installed, with the same script, if I select to run on GTX2080ti, the error occurs, and if I run on GTX1080ti, it works right

~/Molecule_Optimizer.py in forward(self, atom_list, bond_list, atom_degree_list, bond_degree_list, atom_mask)
     33         atom_mask = atom_mask.unsqueeze(2)
     34         batch_size,mol_length,num_atom_feat = atom_list.size()
---> 35         atom_feature = self.dropout(self.atom_fc(atom_list))
     36 
     37         bond_neighbor = [bond_list[i][bond_degree_list[i]] for i in range(batch_size)]

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     53 
     54     def forward(self, input):
---> 55         return F.linear(input, self.weight, self.bias)
     56 
     57     def extra_repr(self):

~/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1024         return torch.addmm(bias, input, weight.t())
   1025 
-> 1026     output = input.matmul(weight.t())
   1027     if bias is not None:
   1028         output += bias

RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1533672544752/work/aten/src/THC/THCBlas.cu:249
1 Like
3

I also encountered the same problem. How did you solve it?

4

I also occurred this question ,finally i find that it is fc

5

Excuse me, could you please tell me how to tackle this problem? I found that it will occur when I carry out the “loss.backward()”.

6

Is this resolved? It may be related to an issue I am having and cannot resolve. See here