I am adding extra vocabulary tokens to embedding layer in T5 model on huggingface repo. During fine tuning , I do not want any updates on pre trained embeddings but only on added extra vocabulary tokens. I am using register_hook to set gradients to 0 on pre trained embeddings. However while fine tuning all embeddings are updated.Colab notebook showing that in simple cases register_hook on slice of embedding works
However does not work on T5
