MNIST normalization and torchvision's Normalize

#1

I want to normalize the MNIST dataset.
Here is how I calculate mean and standard-deviation:

transform=tv.transforms.Compose([tv.transforms.ToTensor()])
train_dataset = tv.datasets.MNIST('../data', train=True, download=True, transform=transform)
mean = torch.mean(torch.Tensor.float(train_dataset.data))
std = torch.std(torch.Tensor.float(train_dataset.data))

If I manually normalize the data like this:

train_dataset.data = (train_dataset.data - mean) / std
test_dataset.data = (test_dataset.data - mean) / std

I get decent accuracy (~0.978), though not better than without normalization (~0.9796).

However, if I use the Normalize transform with the same mean and std:

transform=tv.transforms.Compose([tv.transforms.ToTensor(), tv.transforms.Normalize(mean, std)])
train_dataset = tv.datasets.MNIST(’…/data’, train=True, download=True, transform=transform)
test_dataset = tv.datasets.MNIST(’…/data’, train=False, download=True, transform=transform)

I get very low accuracy (0.135). Why is that, and how should I use Normalize instead?

Second question: I also tried (manual) pixel-wise normalization:

px_mean = torch.mean(torch.Tensor.float(train_dataset.data), dim=0)
px_std = torch.std(torch.Tensor.float(train_dataset.data), dim=0)+1e-10
train_dataset.data = (train_dataset.data-px_mean)/px_std
test_dataset.data = (test_dataset.data-px_mean)/px_std

but this, too, gives me very low accuracy (~0.135). Am I doing it wrong, and if so, how to do it correctly?

#2

The internal .data will store the raw dataset in uint8 with values in the range [0, 255].
The mean of these values (transformed to FloatTensors) would thus be 33.3184.
Normalizing the raw data with these values would thus work.
However, since ToTensor() already normalizes the tensors to the range [0, 1], the mean and std in transforms.Normalize should also be in this range (divided by 255.).

1 Like
#3

I see. I tried

transform = torchvision.transforms.Compose([
torchvision.transforms.ToTensor(),
torchvision.transforms.Normalize(mean=mean/255, std=std/255)])

with

mean = 33.3184
std = 78.5675

but this gives me lower accuracy (0.88) than ToTensor() on its own (0.98). Is that expected for MNIST?

#4

It depends on your model and overall training, i.e.:

  • is this result giving the training, evaluation, or final test accuracy
  • how reproducible is this result (i.e. are you always seeing the difference in accuracy using different seeds)
  • how much hyper-parameter turning did you apply (learning rate, model architecture, etc.) or did you use a “static” setup?