nn.InstanceNorm2d in v1.0 and observed that the output tensor of it does not have unit variance. Below, I wrote the test code for both standardization and instance normalization. The
nn.InstanceNorm2d with the
affine=False argument should return the output with channel-wise unit variance.
In addition, the
nn.InstanceNorm2d does not raise an error even if the dimensions of the input do not match.
nn.BatchNorm2d raise an error if the dimensions of the input do not match. Is this intended?
def standardize(x, eps=1e-6): N, C, H, W = x.size() x = x.view(N, C, H*W) mean = torch.mean(x, dim=2, keepdim=True) std = torch.std(x, dim=2, keepdim=True) out = (x - mean) / (std + eps) # (N, C, H*W) return out # Test with standardization x = torch.rand(1, 2, 3, 3) out = standardize(x) print('var: ', torch.var(out, dim=2)) # [1.0, 1.0] # Test with InstanceNorm2d norm = nn.InstanceNorm2d(2, affine=False) out = norm(x) N, C, H, W = out.size() out = out.view(N, C, H*W) print('var: ', torch.var(out, dim=2)) # [1.1248, 1.1249] # Dimension not matched norm = nn.InstanceNorm2d(444, affine=False) x = torch.randn(2, 3, 3, 3) out = norm(x) # This does not raise an error norm = nn.BatchNorm2d(444, affine=False) x = torch.randn(2, 3, 3, 3) out = norm(x) # This raise an error