How can I use resnet50 PyTorch model as a Siamese network

Hi all, I use contrastive loss to train resnet50 in form of a Siamese network on CIFAR-10 dataset.
Siamese networks gets two images as input and the here I get two logits of the network as output1 and output2. They would be compared with each other using contrastive loss.
the truth labels are 0 and 1 and I set the last linear of resnet50 to 10 neurons as my logits layer could be had any number of neurons.

net = models.resnet50(pretrained=False)
net.fc = nn.Linear(net.fc.in_features, 10)

my contrastive loss is as below:

class ContrastiveLoss(nn.Module):
    def __init__(self, margin=1.):
        super(ContrastiveLoss, self).__init__()
        self.margin = margin
        self.eps = 1e-9

    def forward(self, output1, output2, target):
        distances = (output2 - output1).pow(2).sum(1)
        losses = 0.5 * (target.float() * distances +
                        (1 + -1 * target).float() * F.relu(self.margin - (distances + self.eps).sqrt()).pow(2))
        return losses.mean()

where output1 and output2 are the output of resnet50 network.
Accuracy would be calculated as follows for any images. Total accuracy is a averaged accuracies of the two logits of images.

def accuracy_contrastiveLoss(y_true, y_prob):
    y_prob = np.where(y_prob.cpu().numpy() <= 0.5, 0, y_prob.cpu().numpy())
    y_prob = np.where(y_prob > 0.5, 1, y_prob)
    y_prob = torch.tensor(y_prob).to('cuda')
    return torch.sum(y_prob == y_true).item()

unfortunately my accuracy didn’t increase more than 53%. what is the problem in this scenario?
any help would be appreciated.