Classifier predicting wrong label on single sentence during inference time

nlp
1

I designed a classifier for the customer review dataset. I am getting high training as well as test accuracy. But when I am using the pre-trained model for classifying a single sentence (checked a sentence from both training and test set), the model is predicting the wrong answer with high probability.
This is my code-

def inference(text):
  doc_idx = [fields[0][1].vocab.stoi[token] for token in text]
  doc_idx_tensor = torch.tensor([doc_idx]).cuda()
  len_tensor = torch.tensor([len(doc_idx)])
  with torch.no_grad():
    model = torch.load(PATH)
    model.eval()
    prediction = model(doc_idx_tensor, len_tensor)
    label = torch.argmax(prediction)
    print(prediction , label)
  return

I don’t know what to do. Any help will be highly appreciated. Thanks in advance.

Here is an example -
text - ‘the product is not good’

output - [[-9.544 17.824]] tensor(1, device='cuda:0')
2

@Soumya_Jain can you check if during the training your label encoding is such that 1 is negative and 0 is positive? Since the training and test accuracy are high, then I’m guessing this might be the issue.

3

@asvskartheek, Thanks for the reply.
I checked it. During training, my label encoding is 0 for negative and 1 for positive only.

4

Is this the only example for which there is a misclassification? This can be the case because, at a fundamental level, ML models are probabilistic; even a 99% test accuracy model can fail over a trivial data point. I would suggest you test this over other examples (try something like 20-30 if manual); if it is working well on those examples, it is acceptable for a model to make this mistake. If the majority of the cases are misclassification, then I guess you have to try different architectures to design a better model.

5

Just curious, what is the pre-trained model you are using? I would like to see the last layer (which can confirm your ‘high’ values are perhaps what is expected and your approach to take argmax is right).
It could also be you may need to fine-tune the ‘head’ of the model, to classify better?

6

@asvskartheek
No, this is happening for almost all examples.

7

@KarthikR

This is how I am comparing true and predicted labels-

num_corrects = (torch.max(predictions_batch, 1)[1].view(target.size()).data == target.data).float().sum()

This is the last layer-

 h = self.fc(self.dropout(h))

I am saving the model with the best validation accuracy and then using that model for classifying single sentences.

8

ok, what is the pre-trained model and are you adding layers to the end of the model? Can you share the model architecture?