i implement instance norm by pytorch basic operations. but the result is different from torch.nn.InstanceNorm2d. Can anyone help me out? Below is my code:

##########################

import torch

import numpy as np

x = torch.rand((8, 16, 32, 32))

a = torch.nn.InstanceNorm2d(256)

a.eval()

with torch.no_grad():

b = a(x)

x_mean = torch.mean(x, axis=(2,3), keepdims=True)

x_var = torch.var(x, axis=(2,3), keepdims=True)

x_norm = (x - x_mean) / torch.sqrt(x_var + 1e-5)

b_numpy = b.numpy()

x_norm_numpy = x_norm.numpy()

# check if b_numpy and x_norm_numpy close to the torlerance of 1e-3

print(np.allclose(b_numpy, x_norm_numpy, atol=1e-3))

# check if b_numpy and x_norm_numpy close to the torlerance of 1e-3

print(np.allclose(b_numpy, x_norm_numpy, atol=1e-4))

##########################

result:

True

False

##########################

so, the result shows that when precision comes to 1e-4, they are different. i don’t know why. can anyone help me to get a more close result to torch.nn.InstanceNorm2d? Thanks in advance!

BTW. The reason why I do not use formula gamma * x_normalized_numpy + beta in the paper is i find that when the first initialization of torch.nn.InstanceNorm2d, all gamma is initialized to [1.0, 1.0, 1.0, …] and all beta is initialized to [0.0, 0.0, 0.0,…]. So, under this condition, x_normalized_numpy = gamma * x_normalized_numpy + beta