Why nn.Embedding results keep changing?

I found the results of nn.Embeding is non-deterministic:

nn.Embedding(3, 2)(Variable(torch.LongTensor([0, 1, 2])))
Variable containing:
-0.3836 -0.2593
-2.2071 -0.5127
-0.3338 -0.8781
[torch.FloatTensor of size 3x2]

nn.Embedding(3, 2)(Variable(torch.LongTensor([0, 1, 2])))
Variable containing:
-0.9080 0.4435
0.1588 -0.4792
0.7375 0.5644
[torch.FloatTensor of size 3x2]

I’ve tried random seeding with:
random.seed(0)
np.random.seed(0)
torch.manual_seed(0)

But the results aren’t different. Aren’t they supposed to be deterministic?

If you set the random seed you do get the same output

>>> torch.manual_seed(5)
>>> torch.nn.Embedding(3,2)(torch.autograd.Variable(torch.LongTensor([0,1,2])))
Variable containing:
-0.4868 -0.6038
-0.5581  0.6675
-0.1974  1.9428
[torch.FloatTensor of size 3x2]

>>> torch.manual_seed(5)
>>> torch.nn.Embedding(3,2)(torch.autograd.Variable(torch.LongTensor([0,1,2])))
Variable containing:
-0.4868 -0.6038
-0.5581  0.6675
-0.1974  1.9428
[torch.FloatTensor of size 3x2]

You are initializing multiple embedding instance/module/matrices. The results are expected to be different.
try

embedding = nn.Embedding(3, 2)
print(embedding(Variable(torch.LongTensor([0, 1, 2]))))
print(embedding(Variable(torch.LongTensor([0, 1, 2]))))
2 Likes

I did initialize multiple embedding instances.
That is intentional code to simulate save & load trained model.

Then you should look at @Diego 's reply above.

@Diego

That resolved my problem! Thank you.

So, It looks like I need to do random seeding every time I create embeddings…

1 Like

No, why? If you just set a seed, then all following operations will be deterministic. So setting at the top your program is fine.

If you want the same seed for every embedding then you do need to do random seed before each embedding since the random seed changes over time.

>>> torch.manual_seed(5)
>>> torch.nn.Embedding(3,2)(torch.autograd.Variable(torch.LongTensor([0,1,2])))
Variable containing:
-0.4868 -0.6038
-0.5581  0.6675
-0.1974  1.9428
[torch.FloatTensor of size 3x2]

>>> torch.nn.Embedding(3,2)(torch.autograd.Variable(torch.LongTensor([0,1,2])))
Variable containing:
-1.4017 -0.7626
 0.6312 -0.8991
-0.5578  0.6907
[torch.FloatTensor of size 3x2]

These don’t output the same since the seed changes overtime during the execution of the program. But it you run the program again with the same seed you will get the exact same output for each tensor respectively.

Now if you to want to have the same output during the whole execution of the program using the same seed then you need to instantiate the embedding like on @SimonW reply. Like so:

>>> torch.manual_seed(5)
>>> a = torch.nn.Embedding(3,2)
>>> a(torch.autograd.Variable(torch.LongTensor([0,1,2])))
Variable containing:
-0.4868 -0.6038
-0.5581  0.6675
-0.1974  1.9428
[torch.FloatTensor of size 3x2]

>>> a(torch.autograd.Variable(torch.LongTensor([0,1,2])))
Variable containing:
-0.4868 -0.6038
-0.5581  0.6675
-0.1974  1.9428
[torch.FloatTensor of size 3x2]

Although I do not know what the purpose of doing the first option will be.

1 Like

That explains all. Thank you!

1 Like

When you save & load the trained model the weights of the embedding layer should be saved & loaded along with all the other model parameters.

So, when loading a saved model you shouldn’t need to fix the random seed in order to get the same embedding as during training.

1 Like

Do you mean I can’t have the same embedding results only by fixing random seed?
Simple examples look like working with @Diego 's suggestion.

Why do I have to persist all embedding layer weights?
First of all, Is an embedding instance an object of learning?

Sure you can. What I’m saying is that you don’t need to fix the random seed when loading a saved model.

The embedding is a learned thing. From the docs, an instance of nn.Embedding has a .weight parameter which contains “the learnable weights of the module of shape (num_embeddings, embedding_dim)”.

1 Like

Wow, I didn’t recognize an embedding instance is a learnable object.

I thought fixing random seed is handy, but there might be some cases where it’s not appropriate… In that case, I will persist embedding weights as you suggest.

Thank you!