I found the result of torch.nn.LayerNorm equals torch.nn.InstanceNorm1d, why?
batch_size, seq_size, dim = 2, 3, 4 x = torch.randn(batch_size, seq_size, dim) #layer norm layer_norm = torch.nn.LayerNorm(dim, elementwise_affine=False) print('y_layer_norm: ', layer_norm(x)) print('=' * 30) # custom instance norm eps: float = 0.00001 mean = torch.mean(x, dim=-1, keepdim=True) var = var = torch.var(x, dim=-1, keepdim=True, unbiased=False) print('y_custom: ', (x - mean) / torch.sqrt(var + eps)) print('=' * 30) # instance norm instance_norm = torch.nn.InstanceNorm1d(dim, affine=False) print('y_instance_norm', instance_norm(x)) print('=' * 30) # follow the description in https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html # For example, if normalized_shape is (3, 5) (a 2-dimensional shape), # the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. input.mean((-2, -1))) mean = torch.mean(x, dim=(-2, -1), keepdim=True) var = torch.var(x, dim=(-2, -1), keepdim=True, unbiased=False) print('y_custom1: ', (x - mean) / torch.sqrt(var + esp)) print('=' * 30)
In the results, I found LayerNorm equals InstanceNorm1d, and I custom the compute progress also found that the description in LayerNorm doc maybe not correct? Do I miss something or LayerNorm and InstanceNorm1d in pytorch are absolutely equal?
Hope someone can answer this question, thanks!