Hello, I’m trying to implement DataLoader for batch processing in a CNN LSTM NLP model.
This is my first time using Pytorch so I’m feeling quite lost and will appreciate some direction.
Suppose my training loop looks like this:
for epoch in range(n_epochs):
epoch_loss = 0.0
interval_loss = 0.0
for iter, train_data in enumerate(train_loader):
words, labels, chars, word_lengths = train.encode(train_data, cuda=cuda)
optimizer.zero_grad()
preds = model(words, chars, word_lengths)
loss = F.cross_entropy(preds, labels)
loss.backward()
optimizer.step()
epoch_loss += loss.item()
interval_loss += loss.item()
Originally the train.encode function looks like this (in which it takes in only a sentence at one time):
def encode(self, i, cuda=True):
# encode the sentence at index i
sentence = self.corpus_original[i]
sentence_lowered = self.corpus_lowered[i]
sentence_tags = self.tags[i]
words = torch.zeros(len(sentence), dtype=torch.long)
labels = torch.zeros(len(sentence), dtype=torch.long)
chars = torch.zeros(len(sentence), self.max_word_len,
dtype=torch.long)
word_lengths = np.zeros(len(sentence), dtype=int) # change to tensor if needed
# Workonthis - can add in the non-lowered words for traning for RNN. Hence it will be
# words_lowered and words_original
for j in range(len(sentence)):
labels[j] = self.tag2idx[sentence_tags[j]]
words[j] = self.word2idx[sentence_lowered[j]]
word_lengths[j] = len(sentence[j])
for k, c in enumerate(sentence[j]): # for each character
chars[j,k] = self.char2idx[c]
if cuda:
words = words.cuda()
labels = labels.cuda()
chars = chars.cuda()
return words, labels, chars, word_lengths
My purpose now is to take in many sentences at once, for example, 32 sentences. My question is will the DataLoader output a tensor of 32 layers, where each layer represents a sentence? If so, how should I write the encode function so that I am utilising multi-core?
I apologise if my questions are very amateurish, I’ve tried my best to read other implementation and the documentation but I’m still feeling hopelessly lost.