Hi,
I have designed a regression model made up of CNN and LSTM. In the forward pass, first I put CNN and use its output for LSTM. Then I compute the score.
The problem is that, after the third epoch, LOSS remains constant. I tested different loss functions(L1Loss.MSEloss, … ), and even optimizers such as AdamW, adam, …, but there is no difference.
Moreover, I checked both output and the Last Hidden Layer of my LSTM to compute the score. Again, I couldn’t see any significant changes.
Herein lies my code.
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
print("Training for %d epochs..." % N_EPOCHS)
for epoch in range(1, N_EPOCHS + 1):
for i, (xA, xB, score) in enumerate(train_loader2, 1):
predicted_score = model(xA, xB)
optimizer.zero_grad()
loss = criterion(predicted_score, score)
total_loss += loss.item()
loss.backward()
optimizer.step()
if i % 20 == 0:
print('[{}] Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.2f}'.format(
time_since(start), epoch, i *
len(xA), len(train_loader.dataset),
100. * i * len(xA) / len(train_loader.dataset),
total_loss / i * len(xA)))
Given the code that I see I can only try to give some pointers that might help. Given that you posted in NLP and use embeddings, I assume that you work with text.
You’re unsqueezing your embeddings from 3d to 4d, so I assume that you use a nn.Conv2d. What are the kernel sizes you choose. Note that you could also use nn.Conv1d which would make the unsqueezing obsolete
To be honest, I have no idea what the expand() methods really does or why you need it. But any method that is needed to “fix” the shape of tensors needs to be applied with great care.
Your setup looks a bit like a Siamese Network. However, you use 2 different CNN layers and 2 different RNN layers – only 1 embedding layer though. In a Siamese setting, you would push both inputs through the same layer.
I trust that you define your LSTMs with batch_first=True
Why do you transpose xA and xB and permute the first 2 dimensions after the embedding layer. I’m pretty sure you can simply remove both .t() and both permute() operations.
hi again Chris,
First of all, thanks a bunch for your tips.
As for your first hint, I used nn.Conv2d since I decided to add more channels for the next test.
*OMG, it’s a typo. EXPAND method was just a trial, I shouldn’t have sent it.
*Yeah, you are right. and also batch_first=True.
In respect of permuting and transposing, you’re definitely right. I wanted to exactly prepare the same shape as the torch example for the embedding layer. (I had thought that it might be the problem)
However, having said that, I don’t know what is wrong. One more thing, I have a varied size of sentences so I used pad sequences to fix the size of each input (It was the same thing as previous papers). I couldn’t use pack_padded due to network producing higher Loss. perhaps it’s because of the differences in the length of two sentences as one input each time.
Hi again Chris,
First of all, thanks a bunch for your tips.
As for your first hint, I used nn.Conv2d since I decided to add more channels for the next test.
OMG, it’s a typo. EXPAND method was just a trial, I shouldn’t have sent it.
Yeah, you are right. and also batch_first=True.
In respect of permuting and transposing, you’re definitely right. I wanted to exactly prepare the same shape as the torch example for the embedding layer . (I had thought that it might be the problem)
However, having said that, I don’t know what is wrong. One more thing, I have a varied size of sentences so I used pad sequences to fix the size of each input (It was the same thing as previous papers). I couldn’t use pack_padded due to network producing higher Loss. perhaps it’s because of the differences in the length of two sentences as one input each time.
And after a while, this appeared.
RuntimeError: Boolean value of Tensor with more than one value is ambiguous.
I don’t know why that happens and which boolean it does mean.
padding is (2,0) for maintaining the height. Besides, each time I considered five rows(words). I’ve changed the shape of it for example (2,100), (28,1),… and even symmetric forms (5,5), … . I wanna check another dataset, simpler than what I have now. Any suggestion is welcomed.
OK, that was I was looking for. If you use nn.Conv2d over sentences, e.g., sequences of word vectors, then there are constraints on the kernel size: if your embedding vectors have as size of embed_size, then kernel_size=(x,embed_size) where x is the only value you can adjust. The issue is that it makes no semantic sense to convolve over the embedding dimension; see this older post of mine.
So if your 100 is the size of your embedding vectors, then kernel_size=(5,100) is valid while kernel_size=(5,5) is not.
By the way, what happens when you give the embeddings directly to the RNN(s), thus skipping the CNN part. I can’t intuitively tell what the CNN layer(s) are suppose to do.
. (I had thought that it might be the problem)