I’m training a CNN for text classification ( https://www.aclweb.org/anthology/D14-1181.pdf). When I try updating 300d pretrained embeddings, I get an OOM error when back propagating (on calling optim.step()) regardless of batch size. However, if the embeddings are frozen or the embeddings are learned there are no issues. I suspect this is due to the way they are being loaded. Below is the embedding loading code.
def _loadEmbeddings(self, vecFilePath): # import pdb;pdb.set_trace() print("Loading pretrained embeddings: ", vecFilePath) with open(vecFilePath, encoding="utf8") as f: readLen = self.vocab.MAX_VOCAB_SIZE if readLen == -1: lines = [line for line in f.readlines()] else: lines = [next(f) for x in range(readLen)] words = [line.split() for line in lines] self.vocab = Vocab(MAX_VOCAB_SIZE = self.vocab.MAX_VOCAB_SIZE, MAX_IDX = 1, UNK_IDX=1) self.vocab.createVocab(words) embeddings = np.zeros((len(self.vocab), self.embedding_dim)) # zeros or rand? for line in lines: values = line.split() word = values index = self.vocab(word) if index: vector = np.array(values[1:], dtype='float32') embeddings[index] = vector embeddings = np.mean(embeddings[2:], axis=0) return nn.Embedding.from_pretrained(torch.from_numpy(embeddings))
Does anyone have any ideas as to possible causes or tools that could help debug this issue?