# How can I get the mean and variance from a saved CNN model? **URL:** \ **Category:** Uncategorized\ **Created:** [January 20, 2023, 1:01pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734 "2023-01-20T13:01:28Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 20, 2023, 1:01pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/1 "2023-01-20T13:01:28Z")
Hi, How can I get the mean, variance, gamma and beta for batchnorm from a saved CNN model? Also, how can I use them for inference? This is my architecture: ```auto ---------------------------------------------------------------- Layer (type) Output Shape Param # ================================================================ Conv2d-1 [-1, 32, 32, 32] 896 BatchNorm2d-2 [-1, 32, 32, 32] 64 ReLU-3 [-1, 32, 32, 32] 0 Dropout-4 [-1, 32, 32, 32] 0 Conv2d-5 [-1, 64, 32, 32] 18,496 BatchNorm2d-6 [-1, 64, 32, 32] 128 ReLU-7 [-1, 64, 32, 32] 0 AvgPool2d-8 [-1, 64, 16, 16] 0 Conv2d-9 [-1, 128, 16, 16] 73,856 BatchNorm2d-10 [-1, 128, 16, 16] 256 ReLU-11 [-1, 128, 16, 16] 0 Dropout-12 [-1, 128, 16, 16] 0 Conv2d-13 [-1, 128, 16, 16] 147,584 BatchNorm2d-14 [-1, 128, 16, 16] 256 ReLU-15 [-1, 128, 16, 16] 0 AvgPool2d-16 [-1, 128, 8, 8] 0 Conv2d-17 [-1, 256, 8, 8] 295,168 BatchNorm2d-18 [-1, 256, 8, 8] 512 ReLU-19 [-1, 256, 8, 8] 0 Dropout-20 [-1, 256, 8, 8] 0 Conv2d-21 [-1, 256, 8, 8] 590,080 BatchNorm2d-22 [-1, 256, 8, 8] 512 ReLU-23 [-1, 256, 8, 8] 0 AvgPool2d-24 [-1, 256, 4, 4] 0 Flatten-25 [-1, 4096] 0 Linear-26 [-1, 32] 131,104 ReLU-27 [-1, 32] 0 Linear-28 [-1, 10] 330 ================================================================ Total params: 1,259,242 Trainable params: 1,259,242 Non-trainable params: 0 ---------------------------------------------------------------- Input size (MB): 0.01 Forward/backward pass size (MB): 5.38 Params size (MB): 4.80 Estimated Total Size (MB): 10.19 ---------------------------------------------------------------- ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 20, 2023, 9:21pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/2 "2023-01-20T21:21:28Z")
You can access the affine, trainable parameters via `.weight` and `.bias` and the running stats via `.running_mean` and `.running_var`. ---
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 25, 2023, 9:34am UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/3 "2023-01-25T09:34:01Z")
Thanks for your response. That worked! I have a saved pytorch CNN model. While doing the inference, I want to know, what is the exact input and output of the batch normalization layers? This is my architecture: ```auto class Cifar10CnnModel(ImageClassificationBase): def __init__ (self): super(). __init__ () self.network = nn.Sequential( # Conv-1 nn.Conv2d(3, 32, kernel_size=3, padding=1), nn.BatchNorm2d(32), nn.ReLU(), nn.Dropout(0.25), # Conv-2 nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(64), nn.ReLU(), nn.AvgPool2d(2, 2), # output: 64 x 16 x 16 # Conv-3 nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(128), nn.ReLU(), nn.Dropout(0.25), # Conv-4 nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(128), nn.ReLU(), nn.AvgPool2d(2, 2), # output: 128 x 8 x 8 # Conv-5 nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(256), nn.ReLU(), nn.Dropout(0.25), # Conv-6 nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(256), nn.ReLU(), nn.AvgPool2d(2, 2), # output: 256 x 4 x 4 nn.Flatten(), nn.Linear(256*4*4, 32), nn.Dropout(0.25), nn.ReLU(), nn.Linear(32, 10)) def forward(self, xb): return self.network(xb) ``` I want to carry out a verification process that the running mean, running variance, gamma and beta are correct by calculating manually. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 25, 2023, 9:56am UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/4 "2023-01-25T09:56:51Z")
Check [this manual implementation](https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py) which shows how the layer execution is performed internally. ---
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 25, 2023, 10:41am UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/5 "2023-01-25T10:41:15Z")
Thanks for your response. Do you have any [similiar](https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6) thing by using which I can get the input and output for batchnorm layers? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 25, 2023, 7:18pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/6 "2023-01-25T19:18:20Z")
Yes, forward hooks should work with my manual module implementation, too, and you should be able to reuse the provided code. ---
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 26, 2023, 6:00pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/7 "2023-01-26T18:00:15Z")
Thank you. I was able to solve my solution and have got the output of each layer. I am also trying to verify the values manually and after doing that I can see that the precision is not matching. For eg. I tried calculating the values for batch normalization layer: ![image](https://discuss.pytorch.org/uploads/default/original/3X/a/0/a006ccf35d32c458e46c203428e9a8dce825a3cc.png) How can I get the exact precison? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 26, 2023, 9:54pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/8 "2023-01-26T21:54:39Z")
You won’t be able to get bitwise-identical results due to the expected errors caused by the limited floating point precision and a potentially different order of operations. Compare the output errors against a small `eps` value or use `allclose` as was done in my code snippet. ---
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 27, 2023, 7:08pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/9 "2023-01-27T19:08:11Z")
Okay, thanks! In batch normalization, will it be ok if I do (modified\_input\*gamma+beta) where modified\_input is ((input - running\_mean)/standard deviation) instead of ((input-running\_mean)/standard deviation)\*gamma+beta? It might seem trivial, but just wanted to confirm once. Basically I want to express the entire batch normalization operation in terms of just addition and multiplication. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 27, 2023, 7:55pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/10 "2023-01-27T19:55:44Z")
I don’t fully understand the difference between these approaches as both apply the same operation. The first one seems to store an intermediate result in the `modified_input` tensor. ---
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 28, 2023, 11:55am UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/11 "2023-01-28T11:55:45Z")
Ok. Also, can you tell me how can I get better precision? As you mentioned, it can’t be exact but possible to get better than the current one? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 28, 2023, 5:19pm UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/12 "2023-01-28T17:19:33Z")
You could use a wider `dtype` such as `float64` for a smaller error. ---
**Author:** ![Saurav\_Pawar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_pawar/32/45962_2.png) [@Saurav\_Pawar](https://discuss.pytorch.org/u/Saurav_Pawar)\ **Post date:** [January 29, 2023, 6:34am UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/13 "2023-01-29T06:34:36Z")
I tried with float64 but still there is a loss of precision. Above you mentioned some reaosns for the loss but are you aware of some more reasons that contributes to the loss in precision. Also, is this dependent on framework, for pytorch the precision is different and for other frameworks it is different? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [January 29, 2023, 6:41am UTC](https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/14 "2023-01-29T06:41:21Z")
Using `float64` would reduce the error you are seeing, but will not reduce it to zero, as it’s expected to see rounding errors using floating point numbers if a different order of operations is used. Note that none of the two values are representing the theoretically “true” value e.g. in this example: ```python x = torch.randn(100, 100, 100) s1 = x.sum() s2 = x.sum(0).sum(0).sum(0) print((s1 - s2).abs()) # tensor(0.0002) ``` as both would suffer from the rounding. [This Wikipedia article](https://en.wikipedia.org/wiki/Single-precision_floating-point_format) describes the numerical format of `float32` in more detail and explains how floating point numbers are represented. The error you are seeing depends on the used algorithms, which is defined by the framework, so I would assume to see a different behavior between PyTorch, TF, etc. unless they call into the same library _and_ a deterministic algorithm.