Expanded embedding is not trainable?

autograd
1

I have a sequence model and I want to concatenate a same piece of small embedding to the input. Specifically, that looks like

if the original embedding is ABCDEFG, the expected one would be

A B C D E F G
X X X X X X X

From a coding aspect, that’s something like:

embed_0_out = embed_0(input_0) # sequence input
embed_1_out = embed_1(input_1) # single input
        
embed_combined_out = torch.cat([embed_0_out, embed_1_out.unsqueeze(1).expand(batch_size, input_0_len, 10)], 2))

However, it seems like the weight for embed_1 is not changing after each iteration, suggesting that it is not trainable. Thus, is there any way to make it trainable?

2

it should work. can you come up with a small runnable example demonstrating the problem?

3

You are right. It is updating only for the row with training data. I was checking the row without even an input so that does not change.

1 Like