Verifying proper usage of bach_norm and eval()

1

I want to make sure that I am utilizing the batchnorm functionality properly: I understand how to code up the layers to perform the batchnorm, however one thing I want to make sure that I am doing right, is basically putting the net into “evaluation” mode, so that the parameters of the batch_norm do not keep getting updated, when I am say, periodically running through the validation set, or when I am done and running over new data etc.

Therefore as I understand it, I have a myNet object, that was made from class Net(nn.module). Now during my training loop, I have to make sure that: myNet.train() is exists, and when I enter my validation loop every so often, I have to execute myNet.eval(), and then run on that. Of course myNet.train() will be executed after that when I go back to training. This also means however that once I have finished the net, I should execute myNet.eval().

Is this correct? Thank you.

1 Like
2

Exactly. This will affect dropout too. The activations at dropout layers need to be scaled during execution to account for the fact that certain units were deactivated during training - eval() takes care of this.

3 Likes