Hello, I am trying to modify the code from here to implement skipgram. I’m having trouble implementing the last layer, where we should have as many softmax’s as we have in our context size. My code works for a context size of 1, since this is just a usual softmax layer, but how do we extend this to a context window larger than 1? Here is my modified code:
CONTEXT_SIZE = 2
EMBEDDING_DIM = 10
# We will use Shakespeare Sonnet 2
test_sentence = """When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a totter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
How much more praise deserv'd thy beauty's use,
If thou couldst answer 'This fair child of mine
Shall sum my count, and make my old excuse,'
Proving his beauty by succession thine!
This were to be new made when thou art old,
And see thy blood warm when thou feel'st it cold.""".split()
# we should tokenize the input, but we will ignore that for now
# build a list of tuples. Each tuple is ([ word_i-2, word_i-1 ], target word)
trigrams = [([test_sentence[i], test_sentence[i+1]], test_sentence[i + 2])
for i in range(len(test_sentence) - 2)]
# print the first 3, just so you can see what they look like
#print(trigrams[:3])
vocab = set(test_sentence)
word_to_ix = {word: i for i, word in enumerate(vocab)}
class MySkipGram(nn.Module):
def __init__(self, vocab_size, embedding_dim, context_size):
super(MySkipGram, self).__init__()
self.embeddings = nn.Embedding(vocab_size, embedding_dim)
self.linear1 = nn.Linear(embedding_dim, 128)
self.linear2 = nn.Linear(128, vocab_size*context_size)
def forward(self, inputs):
embeds = self.embeddings(inputs).view((1, -1))
#out = F.relu(self.linear1(embeds))
out = self.linear1(embeds)
out = self.linear2(out)
log_probs = F.log_softmax(out, dim=1)
return log_probs
losses = []
loss_function = nn.NLLLoss()
model = MySkipGram(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)
optimizer = optim.SGD(model.parameters(), lr=0.001)
for epoch in range(100):
total_loss = 0
for context, target in trigrams:
context_idxs = torch.tensor([word_to_ix[w] for w in context], dtype=torch.long)
target_idx = torch.tensor([word_to_ix[target]], dtype=torch.long)
model.zero_grad()
log_probs = model(target_idx)
loss = loss_function(log_probs, context_idxs)
loss.backward()
optimizer.step()
total_loss += loss.item()
losses.append(total_loss)
print(losses) # The loss decreased every iteration over the training data!
How can I have several softmax layers at the end that work “in parallel”?