Hi, I am using a network to embed some entity into vector space. As the length of the vector decrease during the training. I want to normalize it's length to 1 in the end of each step. Is there any tool that I can use to normalize the embedding vectors?
I think the best thing you can do is to save the embedded indices, and normalize their rows manually after the update (just index_select them, compute row-wise norm, divice, index_copy back into weights). We only support automatic max norm clipping.