Can some please explain how the transforms work and why normalize the data?

#1

Can some please explain how the transforms work and why normalize the data in the mnist dataset?

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, 
                    transform=transforms.Compose([
                        transforms.ToTensor(),
                        transforms.Normalize((0.1307,), (0.3081,))
                    ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
#2

Hi,

You call the transforms in the same way you would for a loss function:

a = np.random.rand(10,10,1)
b = transforms.ToTensor()(a)
b = transforms.Normalize((mean, ), (std, ))(b)

Note the extra parentheses.

#3

Normalizing the data helps the network converge.

If it’s not normalized each image will not be on the same scale, some images will induce bigger errors, other less errors.
Everything error will be added to the gradient with the same weight and backpropagated. Weight corrections will be overestimated for some images and underestimated on others.

At worst your optimizer will not find a way to minimize your loss due to oscillating too much, probably it will converge but will be slower, at best there won’t be any difference.

1 Like