Hello. I’m trying to implement a siamese network with a contrastive loss.
It’s trained on raw tabular data. I’ve started with very a few columns to check if they’re equal (1) or not (0).
The model is:
class Model(nn.Module):
def __init__(self, num_features, num_targets):
super(Model, self).__init__()
self.hidden_size = [5, 5, 5]
self.dropout_value = [0.5, 0.35, 0.25]
self.head = nn.Sequential(
nn.BatchNorm1d(num_features),
nn.Dropout(self.dropout_value[0]),
nn.Linear(num_features, self.hidden_size[0]),
nn.LeakyReLU(),
nn.BatchNorm1d(self.hidden_size[0]),
nn.Dropout(self.dropout_value[1]),
nn.Linear(self.hidden_size[0], self.hidden_size[1]),
nn.LeakyReLU(),
nn.BatchNorm1d(self.hidden_size[1]),
nn.Dropout(self.dropout_value[2]),
nn.utils.weight_norm(nn.Linear(self.hidden_size[1], self.hidden_size[2]))
)
def forward(self, x1, x2):
x1 = self.head(x1)
x2 = self.head(x2)
return x1, x2
And the loss is:
class ContrastiveLoss(nn.Module):
def __init__(self, margin=1.):
super(ContrastiveLoss, self).__init__()
self.margin = margin
self.eps = 1e-9
def forward(self, output1, output2, target, size_average=True):
distances = (output2 - output1).pow(2).sum(1)
losses = 0.5 * (target.float() * distances + \
(1 + -1 * target).float() * F.relu(self.margin - (distances + self.eps).sqrt()).pow(2))
return losses.mean() if size_average else losses.sum()
The problem is that it’s stuck at 0,13x training and 0,2xx validation loss. Using “distances”, it gives some weird results.
For the first 5 samples, instead of:
[1. 0. 0. 1. 1.]
It gives:
[0. , 0. , 0.38202894, 0. , 0.]
What could be the problem? Is this architecture appropriate? Or could it be some technical issues?