# Understanding Instance Normalization 2D with running mean and running var

**URL:** <https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139>\
**Category:** Uncategorized\
**Created:** [February 14, 2022, 4:30pm UTC](https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139 "2022-02-14T16:30:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mypathissional](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mypathissional/32/46672_2.png) [@Mypathissional](https://discuss.pytorch.org/u/Mypathissional)\
**Post date:** [February 14, 2022, 4:30pm UTC](https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139/1 "2022-02-14T16:30:23Z")

</div>

Hi, recently I have been trying to convert StarGAN v1 from Pytorch to ONNX and they had an Instance normalization layer with track\_running\_stats=True. When I exported the model to ONXX it turned out that the exporter does not export the run mean/variance. Nevertheless, the onnx model still gives comparable results to the original model. I was thinking about why it can happen. Then I did the little experiment. I wanted to understand what is the difference between batch norm, instance norm with running mean/var and instance norm without the mean/var. So, I have initialized three layers with the same weights.

```auto
normTrue = nn.InstanceNorm2d(64, affine=True, track_running_stats=True)
normFalse = nn.InstanceNorm2d(64, affine=True)
input = torch.randn(10, 64, 128, 128)
bnorm = nn.BatchNorm2d(64, affine=True, track_running_stats=True)
w = torch.rand(64)
b = torch.rand(64)
m = torch.rand(64)
v = torch.rand(64)
with torch.no_grad():
    normTrue.weight=nn.Parameter(w)
    normTrue.running_mean=m
    normTrue.running_var=v
    normTrue.bias=nn.Parameter(b)
    bnorm.weight=nn.Parameter(w)
    bnorm.running_mean=m
    bnorm.running_var=v
    bnorm.bias=nn.Parameter(b)
    normFalse.weight=nn.Parameter(w)
    normFalse.bias=nn.Parameter(b)

```

It turned out that in the training mode the instance normalization with tracking running stats and without are acting the same.

```auto
with torch.no_grad():
    normoutTrue=normTrue(input).detach().cpu().numpy()
    normoutFalse=normFalse(input).detach().cpu().numpy()
    bnormout=bnorm(input).detach().cpu().numpy()
print(np.max(np.abs(normoutTrue-bnormout)))
print(np.max(np.abs(normoutTrue-normoutFalse)))
print(np.max(np.abs(bnormout-normoutFalse)))

```

0.05608654  
0.0  
0.05608654

But in the inference mode they are different, actually, instance norm with tracking running mean and variance is similiar to the batch norm

```auto
normTrue.eval()
normFalse.eval()
torch.onnx._export(normTrue, # model being run
                           (torch.rand(10,64, 128, 128)),                   
                           "./norm.onnx") ;
bnorm.eval()
torch.onnx._export(bnorm, # model being run
                           (torch.rand(10,64, 128, 128)),                   
                           "./bnorm.onnx") ;

with torch.no_grad():
    normoutTrue=normTrue(input).detach().cpu().numpy()
    normoutFalse = normFalse(input).detach().cpu().numpy()
    bnormout=bnorm(input).detach().cpu().numpy()
print(np.max(np.abs(normoutTrue-bnormout)))
print(np.max(np.abs(normoutTrue-normoutFalse)))
print(np.max(np.abs(bnormout-normoutFalse)))

```

9.536743e-07  
6.2534113  
6.2534113  
So, I have a hard time understanding what is the need for normalizing by run mean/var in the inference stage if they were not used in the training phase and I was actually interested whether ONNX exporter ignores running mean/variances on purpose.

Would be grateful for any clarification,  
Best Regards

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [February 14, 2022, 8:10pm UTC](https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139/2 "2022-02-14T20:10:30Z")

</div>

> [@Mypathissional](#):
>
> l. I was thinking about why it can happen. Then I did the little experiment. I wanted to understand what is the difference between batch norm, ins

Instance norm is implemented using batch norm with a reshaped input (with the batch dimension folded into the channel dimension and an artificial size 1 batch dimension being prepended) and then the running stats (which would be of shaped batch \* channel) are averaged over the “batch” part.  
Once you accept this and use running stats, you can set the batch norm to train or not…

If your instance norm statistics are reasonably stable, the statistics used by the two possible modes “actual input stats” (if running stats is false or the mode is training and “collected running stats” (with running stats in eval mode) should be similar. The latter might be faster in inference because no statistics need to be gathered.

That said, the `track_running_stats=True` is rather not what people expect from instance norm, so I would recommend to not use it unless you have a specific reason to do so.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![Mypathissional](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mypathissional/32/46672_2.png) [@Mypathissional](https://discuss.pytorch.org/u/Mypathissional)\
**Post date:** [February 16, 2022, 12:05pm UTC](https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139/3 "2022-02-16T12:05:34Z")

</div>

@tom Thanks for reaching out. Do I understand correctly that training with track\_running\_stats=True or False is the same, it is only different in the inference stage? Because if so then it will make sense that the outputs during the inference would be of worse quality than in the training mode.

Those are the results from the inference in the training/evalution mode

```auto

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

model= G.main
with torch.no_grad():
    outTrain = model(realInput)
    outTrain=to_numpy(outTrain)[0]
model.eval()
with torch.no_grad():
    outEval = model(realInput)
    outEval=to_numpy(outEval)[0]
    # Postprocess
    outEval=(outEval+1)/2
    outTrain =(outTrain+1)/2
plt.imshow(outTrain.transpose((1,2,0)))
plt.imshow(outEval.transpose((1,2,0)))

```

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a773ed67d3e1d513d48fef9b4b73ea6ce8aa6e0.png)

Then I guess it makes sense that ONNX exporter does not export runmean/runvariance.

---

<div class="post-metadata">

**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\
**Post date:** [February 16, 2022, 12:49pm UTC](https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139/4 "2022-02-16T12:49:52Z")

</div>

> [@Mypathissional](#):
>
> @tom Thanks for reaching out. Do I understand correctly that training with track\_running\_stats=True or False is the same, it is only different in the inference stage? Because if so then it will make sense that the outputs during the inference would be of worse quality than in the training mode.

Yes, this is exactly my understanding as well.

> Then I guess it makes sense that ONNX exporter does not export runmean/runvariance.

indeed.

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![favorxin](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/favorxin/32/53170_2.png) [@favorxin](https://discuss.pytorch.org/u/favorxin)\
**Post date:** [September 13, 2022, 8:26am UTC](https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139/5 "2022-09-13T08:26:42Z")

</div>

Hello! Do you solve the problem that the results of pytorch and onnx are different? I use the InstanceNorm3d and set the track\_running\_stats=False, but the onnx result and pytorch result are different when the results appear two decimal places. I am looking forward to your replay! Thank you!