Error when using CUDA

I got the following error when I’m trying to use CUDA to train a LSTM.

RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

The full code which shows the location:

Traceback (most recent call last):
  File "---/train.py", line 228, in <module>
    main(args)
  File "---/train.py", line 132, in main
    output, _ = model(sample['src_tokens'], sample['src_lengths'], sample['tgt_inputs'])
  File "---\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "---\seq2seq\models\model.py", line 24, in forward
    decoder_out = self.decoder(tgt_inputs, encoder_out)
  File "---\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "---\seq2seq\models\lstm.py", line 303, in forward
    rnn_layer(lstm_input, (tgt_hidden_states[layer_id], tgt_cell_states[layer_id]))
  File "---\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "---\nn\modules\rnn.py", line 974, in forward
    self.bias_ih, self.bias_hh,
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

I’m using

model.to(device)

and

sample = {key: value.to(torch.device("cuda:0")) if type(value) != int else value for key, value in sample.items()}

to make sure the input and the model are devided into the CUDA.

The error happened at the LSTM Decoder:

        self.layers = nn.ModuleList([nn.LSTMCell(
            input_size=hidden_size + embed_dim if layer == 0 else hidden_size,
            hidden_size=hidden_size)
            for layer in range(num_layers)])

Version: 1.7.1+cu101, in Windows 10. :face_with_head_bandage:

The error seems to be raised in:

rnn_layer(lstm_input, (tgt_hidden_states[layer_id], tgt_cell_states[layer_id]))

Could you check the .device attribute of all inputs as well as the parameters of rnn_layer and make sure they are on the GPU?
In case you get stuck, please post an executable code snippet, so that we could reproduce and debug the issue.

2 Likes

Thanks a lot! I found the error, there were two tensors defined without specifying the devices. :innocent: