Embeddings index out of range error

Same problem for me ‘nn.Embedding’ not working

no_layers = 3
vocab_size = padded_rev.shape[0] + 1 #extra 1 for padding
embedding_dim = 400
output_dim = 1
hidden_dim = 256
train_on_gpu = False

model = SentimentRNN(no_layers,seq_length,embedding_dim,hidden_dim,drop_prob=0.5,
                 train_on_gpu=train_on_gpu)
print(model)

SentimentRNN(
  (embedding): Embedding(49975, 400)
  (lstm): LSTM(400, 256, num_layers=3, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (fc): Linear(in_features=256, out_features=1, bias=True)
  (sig): Sigmoid()
)

Unable to train model.It is giving index error.

Could you check the min and max values of the input tensors you are passing to the embedding layer and make sure their values are in the valid range [0, num_embeddings-1]?

CC @mathematics

3 Likes

@ptrblck tx for the response.I figured out my mistake.The value i had given for embedding layer was wrong.Now I made it correct.

Hi @ptrblck,

I’ve found error was on my batch_size function and its internal logic. I was making batches of words given text as
batch_data(words, sequence_length, batch_size)

I was given a problem to Implement the batch_data function to batch words data into chunks of size batch_size using the TensorDataset and DataLoader classes

My illogical code was

def batch_data(words, sequence_length, batch_size):
    batch_size_total = batch_size * sequence_length
    n_batches = len(words) // batch_size_total
    words = words[:n_batches*batch_size_total]
    x = np.arange(len(words)).reshape(batch_size,sequence_length)
    y = x.T[-1] + 1
    feature_tensors = torch.from_numpy(x)
    target_tensors =   torch.from_numpy(y)

    data = TensorDataset(feature_tensors, target_tensors)
    data_loader = torch.utils.data.DataLoader(data,  batch_size=batch_size, shuffle=True)
    return data_loader

I’m trying to solving the problem. Thanks and appretitate for you support :slight_smile: .

edit:
I think i have solved it . I had to change the x values.

@ptrblck
Hi, i have the same problem, but i tried your suggestion and the error released again, can you help me please?
Thanks


max_len=250
min_count=50
batch_size=11


LSTM parameters

embed_size =251
hidden_size = 256
num_layers = 1

training parameters

lr = 0.001
num_epochs = 10


model = LSTMClassifier(embed_size=embed_size,
hidden_size=hidden_size,
vocab_size=vocab_size, # my vocab_size is 67
num_layers=num_layers,
num_classes=train_ds.num_classes,
batch_size=batch_size)

if use_gpu:
model = model.cuda()


criterion = nn.CrossEntropyLoss()
if use_gpu:
criterion = criterion.cuda()

optimizer = optim.Adam(model.parameters(), lr=lr, betas=(0.7, 0.99))
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.975)

`hist = train(model, train_dl, valid_dl, criterion, optimizer, scheduler, num_epochs)`

IndexError                                Traceback (most recent call last)
<ipython-input-192-7e0f888e140e> in <module>
----> 1 hist = train(model, train_dl, valid_dl, criterion, optimizer, scheduler, num_epochs)

~\train_utils.py in train(model, train_dl, valid_dl, criterion, optimizer, scheduler, num_epochs)
     85 
     86         ## perform one epoch of training and validation
---> 87         trn_loss, trn_acc = train_step(model, train_dl, criterion, optimizer, scheduler)
     88         val_loss, val_acc = validate_step(model, valid_dl, criterion)
     89 

~\train_utils.py in train_step(model, train_dl, criterion, optimizer, scheduler)
     29         model.zero_grad()
     30 
---> 31         output = model(train_inputs.t())
     32 
     33         loss = criterion(output, train_labels)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-181-360ed93de0e5> in forward(self, x)
     24 
     25     def forward(self, x):
---> 26         x = self.embedding(x)
     27         x, self.hidden = self.lstm(x, self.hidden)
     28         x = self.fc(x[-1])  # select the last output

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    122 
    123     def forward(self, input: Tensor) -> Tensor:
--> 124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
    126             self.norm_type, self.scale_grad_by_freq, self.sparse)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1812         # remove once script supports set_grad_enabled
   1813         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1814     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1815 
   1816 

IndexError: index out of range in self

Could you print the min and max values for all inputs as well as the embedding size?
Based on the error message you are passing word indices to an embedding layer, which are not in the expected range.

print(len(train_ds))
44

print(len(valid_ds))
22

vocab_size = 2 + len([w for (w, c) in train_ds.vocab.word2count.items() if c >= min_count])
print(vocab_size)
67
i don’t have any other input…
@ptrblck

These values would be interesting to see:

print(train_inputs.min())
print(train_inputs.max())

before using model(train_inputs.t()).

Thanks for your reply, My inputs are as shown below:
@ptrblck

print(train_inputs.size())
print(train_inputs.min())
print(train_inputs.max())
print(train_inputs)

torch.Size([11, 250])
tensor(0)
tensor(8427)
tensor([[ 243, 97, 1999, …, 3, 19, 166],
[ 0, 0, 0, …, 8083, 2, 8084],
[8293, 1146, 2027, …, 2, 8225, 2],
…,
[ 0, 0, 0, …, 2, 597, 598],
[ 4, 4, 4, …, 34, 2019, 5802],
[5735, 6, 5878, …, 26, 10, 228]])

Thanks for the update.
In a previous post you’ve posted: # my vocab_size is 67. If that value is used as num_embeddings in an nn.Embedding module, the error is expected, as your inputs have to stay in the expected range [0, num_embeddings-1], so [0, 66] in your case, while the max. value is 8427.

@ptrblck
Thanks for your attention,
Sorry for my questions, i am new in python and deep learning.
can you help me that this code related to earn vocab_size, is wrong?

vocab_size = 2 + len([w for (w, c) in train_ds.vocab.word2count.items() if c >= min_count])

I should mention that my project is text classification, and i am using persian language.

I’m not familiar with your code, but it seems you are trying to calculate the length of all words, which are used more than min_count times.
I guess the condition might reduce the number of unique words and you are thus running into the error.

The error is generally raised, as you are initializing the embedding lookup table (weight parameter in nn.Embedding) with a specific vocab_size, while the input tensor tries to index this lookup table at an invalid index.
If you want to use word indices of 8427, your nn.Embedding would have to have at least num_embeddings>=8427.

Thanks for your reply and guidance, i will look at my tokenizer, maybe the problem is there.
@ptrblck

@ptrblck PLEASE HELP ME OUT .I dont know from where the error is coming .I was using layoutlm for some tokenclassification.

this is my specification

{
“attention_probs_dropout_prob”: 0.1,
“finetuning_task”: null,
“hidden_act”: “gelu”,
“hidden_dropout_prob”: 0.1,
“hidden_size”: 768,
“initializer_range”: 0.02,
“intermediate_size”: 3072,
“is_decoder”: false,
“layer_norm_eps”: 1e-12,
“max_position_embeddings”: 512,
“max_2d_position_embeddings”: 1024,
“num_attention_heads”: 12,
“num_hidden_layers”: 12,
“num_labels”: 2,
“output_attentions”: false,
“output_hidden_states”: false,
“output_past”: true,
“pruned_heads”: {},
“torchscript”: false,
“type_vocab_size”: 2,
“use_bfloat16”: false,
“vocab_size”: 30522
}

! python run_seq_labeling.py
–data_dir /content/drive/MyDrive/dataset_short/LLM/working/dataset
–labels /content/drive/MyDrive/dataset_short/LLM/working/dataset/labels.txt
–model_name_or_path “{pretrained_model_folder}”
–model_type layoutlm
–max_seq_length 512
–do_lower_case
–do_train
–num_train_epochs 10
–logging_steps 50
–save_steps -1
–output_dir output
–overwrite_output_dir
–per_gpu_train_batch_size 8
–per_gpu_eval_batch_size 16

You are most likely running into the same error as already described in this topic, i.e. your input tensor contains indices which are out-of-bounds for the embedding layer. Did you read my previous answers, which explain which value range is expected?

@ptrblck I’m totally new to it. Can you again say what exactly I should do now please. Any edit that can make it work. Actually it was working fine before for less labels as I increased one label its showing this error.

You have to check the range of the input tensor to the nn.Embedding layer and make sure its values are in [0, num_embeddings-1].
Here is another small example showing the IndexError:

# create embedding layer which expects inputs with indices in [0, num_embeddings-1]
num_embeddings = 10
embedding_dim = 5
emb = nn.Embedding(num_embeddings, embedding_dim)

# valid input since in range [0, 9]
x = torch.tensor([0, 4, 9])
out = emb(x)

# invalid input as it contains indices with are out-of-bounds
x = torch.tensor([0, 4, 10]) # 10 is invalid!
out = emb(x)
# IndexError: index out of range in self

Embeddings use the input to index their weight as a lookup table. If the index is invalid an error is raised.

Embedding layers expect integer inputs, not floats, so I don’t fully understand your statement.

I was getting the error again and again, and after some time I found that I was using the wrong model everytime .You are correct, I think that I need to review my basics. Sorry for the wrong lead