How to normalize a tensor to 0 mean and 1 variance?

4

Thanks but this one won’t work for my use case , as I am not trying to do this when I load the data, but as part of another calculation that I am performing during training.

1 Like
5

Yeah I tried this and I always get an error:

" for t, m, s in zip(tensor, self.mean, self.std):
TypeError: zip argument #2 must support iteration"

6

Oh, the mean and std need to be arrays?

7

so you cant zip self.mean and self.std if they are sinlge values. zip takes multiple iterables and returns packaged tuples.

means = [self.mean] * tensor.size()[0]
stds = [self.std] * tensor.size()[0]
for t, m, s in zip(tensor, means, stds):
  # do stuff

turn the means and stds into a length n array where n is the length of ‘tensor’ or tensors

3 Likes
8

Great! I see. thank you very much.

9

I haven’t figured out how to use transforms.Normalize on input data that is not an image. I get TypeError: tensor is not a torch image. Is there any way to use this method on non-images?

10

Normalize works on tensors, so the error message might come from another transformation:

norm = transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
x = torch.randn(3, 224, 224)
out = norm(x)
1 Like
11

That only works because your tensor has the dimensions of an Image. If you look at the documentation, it says torchvision.transforms.Normalize is used to Normalize a tensor image with mean and standard deviation. The argument is described as a

tensor ( Tensor ) – Tensor image of size (C, H, W) to be normalized.

My data is sequence data of dimension torch.Size([4, 589, 4])

2 Likes
12

Actually, you’re right the error does go away if I get the dimensions right:

norm = transforms.Normalize((30, 30, 30, 30), (20, 25, 30, 35))
x = torch.randn(4, 589, 4)
out = norm(x)

But I don’t think this is applying the normalization correctly. The data from my data loader is shaped [batch_size, seq_length, x_dim] so the scaling should be applied to the last dimension, whereas I think normalize is applying the scaling across the first dimension (the set of image colour maps).

13

Is it possible to extend/apply the transforms.Normalize to normalize multidimensional tensor in custom pytroch dataset class? I have a tensor with shape (S x C x W x H) and I want to normalize on C dimension.

14

thanks, and i have a question on how to set mean and std for each channel, are they calculated from dataset?

15

Yes, you can calculate the mean and std from your training dataset or use some “default” values e.g. from ImatgeNet.

16

Is there a way to apply different transforms to the mask vs input? For example, I want to apply all deformation transforms to both, but i only want to normalize and totensor the predicted masks (not target)

18

I would recommend to use the functional API for these use cases, as it allows you to apply the same “random” transformation on the data and target, and can also be used to call some transformations on one of these tensors separately.
Have a look at this example.

1 Like
19

Me and @FilipAndersson245 found out that the correct way to unnormalize is:

x * std + mean

We also had to clamp a few values outside of [0,1].

For a single image the code would look something like this:

def inv_normalize(img):
    mean = torch.Tensor([0.485, 0.456, 0.406]).unsqueeze(-1)
    std= torch.Tensor([0.229, 0.224, 0.225]).unsqueeze(-1)
    img = (img.view(3, -1) * std + mean).view(img.shape)
    img = img.clamp(0, 1)
    return img

Feel free to help if the code can be written in a simpler way!

1 Like
20

Hi @ptrblck, I am also trying to do transform.Normalize(mean, std) outside data-loader but somewhere in the training process. I am not sure how would I do this for a batch of images.

Also, I am using F.normalize(tensor, p=1, dim=1) inside my model. Now, If I am loading the data with transforms.Normalize(mean, std) does it mean I am applying the same Normalization twice?

I saw the source for transforms.Normalize and it appears to be using F.normalize(tensor, self.mean, self.std, self.inplace) which I am not sure is the same thing or different.

21

To apply transforms.Normalize on a batch you could either run this transformation in a loop on each input or normalize the data tensoe manually via:

x = (x - mean) / std

Inside transforms.Normalize the torchvision.transforms.functional API will be used as F.normalize.
This is not the same methods as torch.nn.functional.normalize and will accept different input arguments.

4 Likes
22

you can do it for any tensor or batch of tensors in 3 steps:

import torch
a = torch.randn(2,3,4, 4) # batch=2, channel=3,H=4,W=4 and I want to normalize across per channel
mu = torch.mean(a,dim=(2,3),keepdim=True)
sd = torch.std(a,dim=(2,3),keepdim=True)
normalized_res = (a - mu)/sd
1 Like
23

From where you get mean and std? Why did you choose these particular values?

24

I’ve used the ImageNet stats, since they were used to pretrain the torchvision.models for classification as described here.