Exponential Moving Average in PyTorch

[Toby]

Hi,
Exponential Moving Average (EMA) is an important feature in state-of-the-art research, in Tensorflow they already implemented it with tf.train.ExponentialMovingAverage. I wonder why the Pytorch team has not released an official version of EMA.

In others related question, there is no expert confirm that it is the correct implementation:

Exponential Moving Average in PyTorch, for weights and gradients

Do we need to apply exponential moving average to weights during training when we use Adam (or other optimizers)? My EMA is defined as: class EMA(object): def __init__(self, mu): self.mu = mu self.shadow = {} def register(self, name, val): self.shadow[name] = val.clone() def __call__(self, name, x): assert name in self.shadow new_average = (1.0 - self.mu) * x + self.mu * self.shadow[name] self.shadow[name] = new_average.clone() return new_average My code is like…

How to apply exponential moving average decay for variables?

I am reading following paper. And it uses EMA decay for variables. BI-DIRECTIONAL ATTENTION FLOW FOR MACHINE COMPREHENSION During training, the moving averages of all weights of the model are maintained with the exponential decay rate of 0.999. They use TensorFlow and I found the related code of EMA. In PyTorch, how do I apply EMA to Variables? In TensorFlow, there is tf.train.ExponentialMovingAverage class. https://www.tensorflow.org/versions/r0.12/api_docs/python/train/moving_avera…

Or in this repo, the issue said there was some problem with the code.