RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2'

nlp
1

I’m trying to implement a skip-gram model. When I pass the input vector to the hidden layer, I get the error:
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 ‘mat2’

ONE_HOT_SIZE  = len(dictionary)
EMBEDDING_DIM = 200
EPOCH_NUM     = 5

def one_hot(index):
    one_hot_array = torch.zeros(ONE_HOT_SIZE, dtype=torch.long)
    one_hot_array[index] = 1.
    return one_hot_array

class SkipGramModel(nn.Module):
    def __init__(self, one_hot_size, embedding_dim):
        super(SkipGramModel, self).__init__()
        self.hidden = nn.Linear(one_hot_size, embedding_dim)
        self.linear_out = nn.Linear(embedding_dim, one_hot_size)
    def forward(self, x):
        embed = self.hidden(x)
        out   = self.linear_out(embed)
        out   = F.log_softmax(out, dim=0)
        return out

model         = SkipGramModel(ONE_HOT_SIZE, EMBEDDING_DIM)
loss_function = nn.NLLLoss()
optimizer     = optim.SGD(model.parameters(), lr=0.01)

losses = []
total_count = EPOCH_NUM * len(targets)

with tqdm(total = total_count) as pbar:
    for epoch in range(EPOCH_NUM):
        total_loss = 0
        for target in targets:
            # Convert input word and target to one-hot vectors
            word   = one_hot(target[0])
            target = one_hot(target[1])

            # Reset gradient
            model.zero_grad()

            # Get the output from the model
            prediction = model(word)
            
            # Calculate loss
            loss = loss_function(prediction, target)

            # Optimize
            loss.backward()
            optimizer.step()

            # Add loss to total loss
            total_loss += loss

            # Update progress bar
            pbar.update(1)
        losses.append(total_loss)

Error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-39-884ed7fa9d4b> in <module>
     11 
     12             # Get the output from the model
---> 13             prediction = model(word)
     14 
     15             # Calculate loss

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

<ipython-input-36-238e5eb99042> in forward(self, x)
      5         self.linear_out = nn.Linear(embedding_dim, one_hot_size)
      6     def forward(self, x):
----> 7         embed = self.hidden(x)
      8         out   = self.linear_out(embed)
      9         out   = F.log_softmax(out, dim=0)

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     90     @weak_script_method
     91     def forward(self, input):
---> 92         return F.linear(input, self.weight, self.bias)
     93 
     94     def extra_repr(self):

c:\users\archan\appdata\local\programs\python\python36\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1406         ret = torch.addmm(bias, input, weight.t())
   1407     else:
-> 1408         output = input.matmul(weight.t())
   1409         if bias is not None:
   1410             output += bias

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2'
2

word is probably a LongTensor, which will yield this error.
You can transform it to a FloatTensor by calling .float() on it:

word = word.float()
prediction = model(word)

Not relevant for this error, but which loss function are you using?

3 Likes
3

I’m using the NLLLoss function. After making the change you suggested, I get the error “Expected two dimensions, got one”

4

Could you print the shape or words?
nn.Linear expects the input as a tensor of shape [batch_size, *, in_feautres].

Also, nn.NLLLoss expects a LongTensor as the target containing the class indices. It should therefore not be one-hot encoded.

1 Like
5

I messed up by feeding the training set one by one. I’ll feed batches instead. Thanks for the help!
And why shouldn’t the vector consist of one-hots?
Shouldn’t each class be true or false instead?

6

No, as given in the docs, the loss expects class indices instead:

The target that this loss expects should be a class index in the range [0,C−1] where C = number of classes; if ignore_index is specified, this loss also accepts this class index (this index may not necessarily be in the class range).

You can see the applied formula in the CrossEntropyLoss docs. Note that nn.CrossEntropyLoss will internally use log_softmax + NLLLoss.

1 Like
7

I read it. I’ll pass the target without encoding it. Thanks a bunch!

1 Like
8

I thought the error stated that it was expecting a LongTensor instead of a FloatTensor. I put .float() to the corresponding place and the error disappeared.
Why is this error message indicating it is actually expecting a FlaotTensor?
Thanks!

9

The RuntimeError claims that the first argument to the operation is the “right” one and the second argument has the wrong data type.
In this case the second argument is the weight as seen in the stack trace, which explains the error message.