How to implement moving average of parameters

1

Hi everyone! I want to implement the model in the paper “Sequence Level Contrastive Learning for Text Summarization”.

1650683619(1)
1650683619(1)755×651 81.9 KB

As the paper mentioned, the architecture of two f is the same, but only updates the parameters θ in fθ, and the parameters ξ in fξ is the moving average of θ.
1650683749(1)

I want to know:

  1. whether or not parameters ξ should be calculated after every backward update the parameters θ
  2. how to write codes to implement the moving average update of parameters ξ after I set require_grad=False for ξ
2

You may find something from here.

Let me know if it works.