How to compute the distance between two word embedding?

Hi, everyone. I am trying to train a model to output an approximate word embedding. I want to make it close the original word embedding.

Which loss function should I use? Is it MSE Loss?

Thanks.

If I understand you have a pre-trained word embedding model. And you have your own model that you want to train so that it produces representations very close to the one produced by the first one.
Is that it?

1 Like

Check CosineSimilarity — PyTorch 1.7.0 documentation

1 Like

Expanding on @Versus’s answer, the loss function form is nn.CosineEmbeddingLoss.

nn.TripletMarginLoss may be helpful to you as well.

1 Like

That’s right :grinning:

Thank you :grinning:

Sorry, I don’t know how to set the y label of nn.CosineEmbeddingLoss in my case. Is it -1?