LayerNorm == InstanceNorm?

[zyh3826]

I found the result of torch.nn.LayerNorm equals torch.nn.InstanceNorm1d, why?
testing code:

batch_size, seq_size, dim = 2, 3, 4
x = torch.randn(batch_size, seq_size, dim)

#layer norm
layer_norm = torch.nn.LayerNorm(dim, elementwise_affine=False)
print('y_layer_norm: ', layer_norm(x))
print('=' * 30)

# custom instance norm
eps: float = 0.00001
mean = torch.mean(x, dim=-1, keepdim=True)
var = var = torch.var(x, dim=-1, keepdim=True, unbiased=False)
print('y_custom: ', (x - mean) / torch.sqrt(var + eps))
print('=' * 30)

# instance norm
instance_norm = torch.nn.InstanceNorm1d(dim, affine=False)
print('y_instance_norm', instance_norm(x))
print('=' * 30)

# follow the description in https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html
# For example, if normalized_shape is (3, 5) (a 2-dimensional shape),
# the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. input.mean((-2, -1)))
mean = torch.mean(x, dim=(-2, -1), keepdim=True)
var = torch.var(x, dim=(-2, -1), keepdim=True, unbiased=False)
print('y_custom1: ', (x - mean) / torch.sqrt(var + esp))
print('=' * 30)

results:

In the results, I found LayerNorm equals InstanceNorm1d, and I custom the compute progress also found that the description in LayerNorm doc maybe not correct? Do I miss something or LayerNorm and InstanceNorm1d in pytorch are absolutely equal?
Hope someone can answer this question, thanks!

[snio89 (Ascii)]

This seems like a bug of nn.InstanceNorm1d when affine=False.

nn.InstanceNorm should warn user if input channel is inconsistent with num_features, even when affine=False

nn.InstanceNorm1d should take an input of the shape (batch_size, dim, seq_size).
However, if affine=False, nn.InstanceNorm1d can take an input of the wrong shape (batch_size, seq_size, dim) and provides a LayerNorm-like result.