RuntimeError: CUDNN Error while running the LSTM

The below mentioned error is produced when I am trying to use LSTM:


File "train_vid_model_xent_htri.py", line 258, in train
    outputs, features = model(imgs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 112, in forward
    return self.module(*inputs[0], kwargs[0])
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, kwargs)
  File "/home/vsl3/Desktop/Multi_camera_fusion_with_attention/Code/training_model_from_scratch/models/ResNet.py", line 48, in forward
    out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (seq_length, batch_size, hidden_size*2)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 323, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 287, in forward
    dropout_ts)
**RuntimeError: CUDNN_STATUS_EXECUTION_FAILED** 

torch.cuda.is_available() gives True value

Before passing to model:
imgs, pids = imgs.cuda()
outputs, features = model(imgs)

When I am making : torch.backends.cudnn.enabled=False

Then the error obtained is:

  
outputs, features = model(imgs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 112, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vsl3/Desktop/Multi_camera_fusion_with_attention/Code/training_model_from_scratch/models/ResNet.py", line 48, in forward
    **out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (seq_length, batch_size, hidden_size*2)**
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/modules/rnn.py", line 192, in forward
    output, hidden = func(input, self.all_weights, hx, batch_sizes)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 323, in forward
    return func(input, *fargs, **fkwargs)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 243, in forward
    nexth, output = func(input, hidden, weight, batch_sizes)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 86, in forward
    hy, output = inner(input, hidden[l], weight[l], batch_sizes)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 115, in forward
    hidden = inner(input[i], hidden, *weight)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 28, in LSTMCell
    hgates = F.linear(hidden[0], w_hh)
  File "/home/vsl3/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 994, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'mat2'

It seems that the nn.LSTM() is not able to take cuda tensors. Unable to resolve this error.

The second error suggests that somewhere you are passing in a CPU tensor where it should be a CUDA tensor.

Have you called model.cuda() to put the LSTM’s weights onto the gpu?

Yes, I have called model.cuda() through below line:

model = nn.DataParallel(model).cuda()

The second error is expecting torch.FloatTensor but torch.cuda.FloatTensor is found.
Don’t know why is it expecting torch.FloatTensor.

Oh interesting, thank you for pointing that out. Are you sure you are sending a CUDA tensor into the model and not a CPU tensor?