Dear All,
I would like to ask for your help! I’ve implemented a model described in this paper but when I test the model I see strange results. That is the loss is always a really small number but the accuracy of the model on the training set is bad and during training it always takes on the same value.
Please find below an example:
Epoch: 1 Loss: 0.00005, Accuracy: 0.36782 Time: 3.7852s
Epoch: 2 Loss: 0.00005, Accuracy: 0.36782 Time: 3.6561s
Epoch: 3 Loss: 0.00005, Accuracy: 0.36782 Time: 3.6510s
…
I know that there might be errors in the code that calculates the accuracy but I would like to ask you to have a look at the implementation of the loss function. I’m fairly new to PyTorch which is why I’m not sure if I’ve done everything correctly. Please find the code snippet below:
class EnergyLoss(nn.Module):
"""
Implements the loss function described in
paper:
'Learning Text Similarity with Siamese Recurrent Networks'
by:
Paul Neculoiu, Maarten Versteegh and Mihai Rotaru
{ neculoiu, versteegh, rotaru}@textkernel.nl
"""
def __init__(self, margin):
"""
Initializes the loss function
:param margin:
"""
super(EnergyLoss, self).__init__()
self.m = margin
def forward(self, cos_sim, y_target):
"""
Calculate loss
Dimensionality:
N: number of examples in the current minibatch
:param cos_sim: vector of cosine-similarities which is of shape (N, 1)
:param y_target: binary vector of target classes which is of shape (N, 1) [1: similar; 0: dissimilar]
:return: energy_loss: loss calculated on the current minibatch
"""
# input check
if len(cos_sim.size()) != 2 or cos_sim.size()[1] != 1:
raise ValueError("The vector of cosine similarities is invalid! Size: (N, 1)")
if len(y_target.size()) != 2 or y_target.size()[1] != 1:
raise ValueError("The vector of target classes is invalid! Size: (N,1)")
# minibatch info
nb_examples = cos_sim.size()[0]
# loss calculation
energy_vec_pos = torch.zeros((nb_examples, 1)).type(float_type)
energy_vec_neg = torch.zeros((nb_examples, 1)).type(float_type)
# for filling the output vector
pos_mask = y_target == 1
neg_mask = y_target == 0
# get corresponding entries
pos_sim = cos_sim[pos_mask].type(float_type)
neg_sim = cos_sim[neg_mask].type(float_type)
# apply defined functions
# similar case
pos_sim = 0.25 * (1. - pos_sim) ** 2
# dissimilar case
neg_sim[neg_sim >= self.m] = 0
neg_sim[neg_sim < self.m] = neg_sim ** 2
# create vectors for both cases
# energy_vec_pos[pos_mask] = pos_sim
energy_vec_pos = Variable(
energy_vec_pos.masked_scatter_(pos_mask.data, pos_sim.data),
requires_grad = True
)
energy_vec_neg = Variable(
energy_vec_neg.masked_scatter_(neg_mask.data, neg_sim.data),
requires_grad = True
)
energy_loss = torch.sum(
y_target.float() * energy_vec_pos.float() + (1. - y_target.float()) * energy_vec_neg.float(),
dim = 0
)
return energy_loss
Thank you very much for your help in advance!