Measuring cosine similarity reconstructed vector

I’m trying to use a convolutional auto-encoder to reconstruct some vectors (not images).
In order to assess whether it’s working I’d like to plot the cosine similarity between the input data and the reconstructed vector. What I’m doing right now is this:

    with torch.no_grad():
        for data in loader_val: # Validation dataset
            # CAE
            data, noise = data # Use data w/ noise as input model
            data = data.unsqueeze(1).float()
            noise = noise.unsqueeze(1).float()
            out = model(noise)
            loss = criterion(out, data)
            # Cosine similarity
            cos = CosineSimilarity()
            cosine_sims.append(cos(data, out))

I wanted to visually see how it changes after each epoch and so I used the following commands to plot it:

plt.plot([np.mean(np.abs(el.numpy())) for el in cosine_sims])

The problem is that the values start at 0.9 and they just fluctuate (not decreasing or increasing). Is there something wrong in the way I use CosineSimilarity?

What kind of criterion are you currently using to train your model?

I’m currently using an MSELoss.

The Euclidean distance does not necessarily correlate with the angle (cosine similarity).
Imagine a point cloud near the origin of a coordinate system. While the MSELoss could be low, the cosine similarity might be high between certain points.

Would it be possible to use the CosineSimilarity as the criterion to train the model?

1 Like

I’ve being thinking about this for a while now. I guess it’s possible but I don’t think I understand how to actually use it. The CosineSimilarity returns a number for each pair of vectors, and not a real loss. Should I take the mean or should I add it to MSELoss? Thanks.

Can a loss function be like:-
Total_loss = MSEloss()+CosineSimilarity()
I think this will be better measure.
Build your custom CosineSimilarityLoss if possible.

I guess, the sign for cosine similarity shall be - (minus). Also, if the vectors are L2 normalized, MSELoss ~ -CosineSimilarity.

Yeah that’s right MSE ~ -(CosineSimilarity) if vectors are l2 normalized.

But I cannot simply add (or subtract) the cosine similarity to the loss function, since the former is not just a real number. Do you recommend taking the mean?

I think MSE should work as MSE ~Approx of (-(CosineSimilarity)).

Given that vectors are l2 normalized.

Why are they L2-normalized?