In a simple test, using the original formula for InstanceNorm2d in here:

eps = 1e-5

N = nn.InstanceNorm2d(1, eps=eps)

x = torch.arange(25).view(1, 1, 5, 5).float()

y1 = N(x)

y2 = (x - x.mean())/torch.sqrt(x.var() + eps)

The error is quite big:

y2-y1

tensor([[[[ 0.0336, 0.0308, 0.0280, 0.0252, 0.0224],

[ 0.0196, 0.0168, 0.0140, 0.0112, 0.0084],

[ 0.0056, 0.0028, 0.0000, -0.0028, -0.0056],

[-0.0084, -0.0112, -0.0140, -0.0168, -0.0196],

[-0.0224, -0.0252, -0.0280, -0.0308, -0.0336]]]])

But the result is consistent. I get the same in CPU and GPU.

What is the right formula for InstanceNorm2d?