How to normalize embedding vectors?

[maplewizard]

Hi, I am using a network to embed some entity into vector space. As the length of the vector decrease during the training. I want to normalize it's length to 1 in the end of each step. Is there any tool that I can use to normalize the embedding vectors?

[apaszke]

I think the best thing you can do is to save the embedded indices, and normalize their rows manually after the update (just index_select them, compute row-wise norm, divice, index_copy back into weights). We only support automatic max norm clipping.

[samarth-robo]

If you want to normalize a vector as a part of a model, this should do it:

assume q is the tensor to be L2 normalized, along dim 1

qn = torch.norm(q, p=2, dim=1).detach()
q = q.div(qn.expand_as(q))

Note the detach(), that is essential for the gradients to work correctly. I'm assuming you want the norm to be treated as a constant while dividing the Tensor with it.

[asdass]

Can I use it to normalise the embedding after each update in the training ?

[samarth-robo]

Yes it could be used to normalize an embedding. I suggest not using the detach() though. I found that it degrades performance.

[asdass]

I see, thanks. But I need to do the constraints on the norms of embeddings( not bigger than 1 ). Do you have any better suggestions to do it?