# How does pytorch’s batch norm know if the forward pass its doing is for inference or training? **URL:** \ **Category:** Uncategorized\ **Created:** [April 24, 2018, 12:08am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857 "2018-04-24T00:08:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 12:08am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/1 "2018-04-24T00:08:48Z")
how does pytorch’s batch norm know if the forward pass its doing is for inference or training? I am evaluating the the test performance of my net but realized that I’m not sure how my net knows if its training phase or inference phase. How does pytorch handle this? ---
**Author:** ![Alpha](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alpha/32/25271_2.png) [@Alpha](https://discuss.pytorch.org/u/Alpha)\ **Post date:** [April 24, 2018, 12:15am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/2 "2018-04-24T00:15:16Z")
Hi, In the source code [here](http://pytorch.org/docs/stable/_modules/torch/nn/modules/batchnorm.html#BatchNorm2d), the function F.batch\_norm has the parameter **self.training** , when you train model, you use **model.train()** when you test, you use **model.eval()** the **model.train()** tells the **self.training** to be True. ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 12:21am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/3 "2018-04-24T00:21:24Z")
I see. I tried running net.train() when there is no batch norm and there was no error thrown…I guess I need to re-do everything cuz I wasn’t using that and I was evalauting test and train at every epoch…so I need to switch around that flag quite often ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 12:50am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/4 "2018-04-24T00:50:07Z")
what are the consequences of having the train mode for both train and inference? ---
**Author:** ![Alpha](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alpha/32/25271_2.png) [@Alpha](https://discuss.pytorch.org/u/Alpha)\ **Post date:** [April 24, 2018, 12:55am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/5 "2018-04-24T00:55:34Z")
It will not be both **train** and **inference**. When use **model.train()**, the **self.training** become **True** , it will keep the **self.training** value until you use **model.eval()** then you use **model.eval()**, the **self.training** become **False** ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 12:56am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/6 "2018-04-24T00:56:07Z")
being in train when evaluating for test data…is what I meant ---
**Author:** ![Alpha](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alpha/32/25271_2.png) [@Alpha](https://discuss.pytorch.org/u/Alpha)\ **Post date:** [April 24, 2018, 1:00am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/7 "2018-04-24T01:00:45Z")
Sorry , I am not exactly understand what you mean. Do you mean that you do training and test alternatively? ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 1:02am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/8 "2018-04-24T01:02:16Z")
What I mean is that I do net.train() and then I do the forward pass on both test and train images. I only train in train images but I didn’t change the flag on test images by accident… ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 1:02am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/9 "2018-04-24T01:02:34Z")
Btw, do you know when are the mean variance for example updated? What happens to them during inference mode? Are they just fixed? ---
**Author:** ![Alpha](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alpha/32/25271_2.png) [@Alpha](https://discuss.pytorch.org/u/Alpha)\ **Post date:** [April 24, 2018, 1:06am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/10 "2018-04-24T01:06:25Z")
I see. As far as I know, the flag self.training just make a difference in the Dropout and Batch Norm layers, so if you don’t use these two types of layer, it may don’t have influence. ---
**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\ **Post date:** [April 24, 2018, 1:07am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/11 "2018-04-24T01:07:14Z")
but I am using batch norm 😂 oooops! ---
**Author:** ![Alpha](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alpha/32/25271_2.png) [@Alpha](https://discuss.pytorch.org/u/Alpha)\ **Post date:** [April 24, 2018, 1:13am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/12 "2018-04-24T01:13:09Z")
It will not keep the mean, variance fixed when **self.training** is true, but the correct way if keep them fixed when evaluating. ---
**Author:** ![lugiavn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lugiavn/32/1799_2.png) [@lugiavn](https://discuss.pytorch.org/u/lugiavn)\ **Post date:** [April 24, 2018, 5:43am UTC](https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857/13 "2018-04-24T05:43:32Z")
I think in the training mode, the batch mean & var are used, the running mean & var are updated but never actually used