Dimension error, GRU model

nlp
1

Hello,

I am getting an index error in the model I made, and I’m not sure how to troubleshoot.
Here is the code:
class NLP_model(nn.Module):
def init(self, vocab_size, embedding_dim, hidden_dim, num_classes):
super(NLP_model, self).init()
self.char_embedding = nn.Embedding(vocab_size, embedding_dim)
self.gru = nn.GRU(embedding_dim, hidden_dim, num_layers = 5, bidirectional= True)
self.fc = nn.Linear(hidden_dim*2,num_classes)
#self.att = nn.MultiheadAttention(embed_dim, num_heads, …)

def forward(self, x):
    x = self.char_embedding(x)
    output, hidden = self.gru(x)
    hidden = torch.cat((hidden[0][-2,:,:], hidden[0][-1,:,:]), dim=1)
    x = self.fc(hidden[0])

    return x

model = NLP_model(len(alphabet), 12, 24, assigned[‘fismaid’].nunique()) #Find the number of fismaids for last variable
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr = 0.001)

for epoch in range(100): #Look into DataLoader for batch processing
y = list()
z = list()
for sentence, label in zip(ls_X_train, ls_y_train): #training_data should be an array of hostnames and labels
model.zero_grad()
output = model(sentence_to_id(sentence)) #sentence is the hostname, label is the fismaid
#print(label)
#print(output.shape)
temp_label = label
label = torch.zeros(assigned[‘fismaid’].nunique())
label[temp_label] = 1.0
#label = torch.tensor(label).unsqueeze(0)
#label = torch.tensor([label]).unsqueeze(1)
#label = torch.tensor(label).unsqueeze(1)
#print(label.shape)
loss = criterion(output, label)
loss.backward()
optimizer.step()
y.append(loss.item())

The error I’m seeing is the following:

IndexError Traceback (most recent call last)
in
8 for sentence, label in zip(ls_X_train, ls_y_train): #training_data should be an array of hostnames and labels
9 model.zero_grad()
—> 10 output = model(sentence_to_id(sentence)) #sentence is the hostname, label is the fismaid
11 #print(label)
12 #print(output.shape)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
887 result = self._slow_forward(*input, **kwargs)
888 else:
→ 889 result = self.forward(*input, **kwargs)
890 for hook in itertools.chain(
891 _global_forward_hooks.values(),

in forward(self, x)
10 x = self.char_embedding(x)
11 output, hidden = self.gru(x)
—> 12 hidden = torch.cat((hidden[0][-2,:,:], hidden[0][-1,:,:]), dim=2)
13 x = self.fc(hidden[0])
14

IndexError: too many indices for tensor of dimension 2

2

Based on the error message it seems you are trying to index too many dimensions.
Assuming hidden has 3 dimensions, then hidden[0] would return a tensor with 2 dimensions only.
In that case, [-2, :, :] won’t work and you would have to remove one indexing dimension.

3

The hidden state of nn.GRU is not a tuple like for nn.LSTM, so hidden[0] is probably not what you want but simply hidden.

Also the shape of hidden is (num_layers * num_directions, batch, hidden_size). I assume your cat() line is there to concatenate the forward in backward direction of the hidden state. If so, you probably want to do the following

# Split num_layers and num_directions (see official docs)
hidden = hidden.view(num_layers, num_directions, batch, hidden_size).
# Consider only the last hidden state (last w.r.t. to num_layer)
hidden = hidden[-1]
# Concatenate forward and backward direction
hidden = torch.cat((hidden[-2,:], hidden[-1,:]), dim=1)

The shape of hidden should now be (batch, 2*hidden_size); otherwise try dim=0, I always get confused how to set this value correctly.

1 Like