LSTM not training when pretrained vectors are loaded using nn.Embedding.from_pretrained

avinash_m · March 16, 2019, 6:00pm

When I load vectors using nn.Embedding.from_pretrained(), training accuracy of the model doesn’t change after every epoch.

But I when initialize them randomly, training accuracy changes.

Here is my code:

gist.github.com

https://gist.github.com/avinashsai/c41c0368653afc58f295fdf76d06902d

lstm.py

import os
import re
import sys
import numpy as np
import pandas as pd
import sklearn
from sklearn.model_selection import train_test_split,StratifiedKFold
from sklearn.metrics import accuracy_score,f1_score

import nltk

This file has been truncated. show original

Here is the data:

Any help is very much appreciated.

SimonW · March 16, 2019, 6:47pm

https://pytorch.org/docs/master/nn.html#torch.nn.Embedding.from_pretrained

see freeze arg

avinash_m · March 16, 2019, 6:50pm

It is true by default

avinash_m · March 17, 2019, 1:14pm

Hi @ptrblck

Can you please help me to solve this problem??

ptrblck · March 17, 2019, 5:37pm

Did you try to set freeze=False as @SimonW suggested?
In case you would like to keep the embedding frozen, could you try to overfit a small data sample (e.g. just 10 samples)? If your model is not able to learn even this small data sample, something else might be wrong with your code.

avinash_m · March 17, 2019, 6:06pm

@ptrblck
problem occurs when I use pretrained word embeddings. If I initialise embeddings randomly using nn.Embedding(vocabsize,embeddingdim) LSTM trains properly.

SimonW · March 17, 2019, 6:42pm

just set freeze=False…

avinash_m · March 17, 2019, 7:09pm

I found out the issue. I didn’t shuffle the data. Hence it is not training. Thank you @ptrblck and @SimonW for your valuable time.