Hi everyone,
Let’s say I finished my model training, and now ready for deployment.
While training my model I used Batch Normalization, Dropout, and L2.
My question is: how do these regularizations deploy?
Do they get dismissed, like delete the layers associated with Batch Normalization and Dropout?
And because the model will be in evaluation mode, so the L2 effect on the loss function will be not computed?
Thank you
For batch normalization, it is a useful transformation even during inference as it is shifting/scaling the outputs of a given layer and the next layer will have learned to depend on this scaling during training. The main difference is that the stats such as mean, variance, etc., do not need to be computed/updated during inference in a deployment setting. Dropout becomes an identity function during deployment inference as none of the hidden units will be dropped during inference. Finally, L2 regularization, as it is computed in the loss function as you note, will not be computed as you do not need to compute the loss function during inference.
Dropout docs describing identity behavior: Dropout — PyTorch 1.13 documentation
The BatchNorm and Dropout changes will happen automatically if you use the builtin implementations in torch.nn and use e.g., model.eval() for deployment.