Mean and Std() for a custom dataset(olivetti)

vision
1

Each of the image has got 64 * 64 dimension. The image is grayscale, so channel =1, I converted the images to TensorDataset which gives me back size of ([1,64,64]) and then i pass this to Dataloader which gives me dimension ([1,1,64,64]) Now if i want to find the mean and std() the following code generates error:

mean = 0.
std = 0.
nb_samples = 0.
for data in loader:
    batch_samples = data.size(0)
   data = data.view(batch_samples, data.size(1), -1)
    mean += data.mean(2).sum(0)
     std += data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

It gives me the following error :

AttributeError: ‘list’ object has no attribute ‘size’

ref: @ptrblck

Now where should i run the loop ? In Dataloader object or in the image itself?

My second question is after i find the mean and std dev what is the code to apply the transorms operation, since it is grayscale , will the code be like the following :

train_transform = transforms.compose(transforms.Normalize ([mean,mean,mean],[std,std,std]))

or should it be like this, as it is a grayscale dataset.

train_transform = transforms.compose(transforms.Normalize ([mean],[std]))

And my third question is how can i apply this mean and std() to my dataset , since it is a custom dataset i will not be able to follow instruction where the datasets are imported from torchvision.

1 Like
2

Your DataLoader might return a list, not a tensor, so you would have to index data before calling size(0) on it.

Also, as mentioned in the thread, the var calculation should yield a better result that the std, since you would now be calculating the average of the stddevs.

You should run this calculation once before your setup of the training procedure.

Since your images are grayscale and will thus have a single channel by default, you should use the second approach.

I don’t understand this question completely.
If you are using a custom Dataset (or e.g. ImageFolder), you would only have to apply the Normalize transformation using your calculated stats.
Could you explain a bit, which instructions you won’t be able to follow?

3

I indexed my list of Tensor as follows :
The Dataloader object has 2 lists. One is the images and another list is the labels. In order to find the mean and standard deviation , i need to go through the batches and have mean and std() for each batch.
Am i indexing it correctly? pls see below. i named it T in the following code. When i print T it gives me back the tensors, But i could not perform T.size on it, it gives me the error,

AttributeError: ‘list’ object has no attribute ‘size’

for data in train_loader:
    T = [data[0]]
print(T)
[tensor([[[[0.3264, 0.1983, 0.2149,  ..., 0.1405, 0.1281, 0.1240],
      [0.2273, 0.2603, 0.2479,  ..., 0.1240, 0.1198, 0.0992],
      [0.2810, 0.2893, 0.2479,  ..., 0.1198, 0.1198, 0.1074],
      ...,
      [0.7975, 0.8182, 0.8058,  ..., 0.2893, 0.1983, 0.0992],
      [0.8058, 0.8347, 0.8017,  ..., 0.3140, 0.2273, 0.1033],
      [0.8223, 0.8430, 0.8099,  ..., 0.3182, 0.2562, 0.1157]]],


    [[[0.4298, 0.4587, 0.5496,  ..., 0.4959, 0.4959, 0.4959],
      [0.4421, 0.4545, 0.5331,  ..., 0.4917, 0.4917, 0.4959],
      [0.4669, 0.4587, 0.5207,  ..., 0.4835, 0.4876, 0.4917],
      ...,
      [0.4421, 0.4339, 0.4380,  ..., 0.3636, 0.3802, 0.3802],
      [0.4421, 0.4339, 0.4380,  ..., 0.3802, 0.3802, 0.3760],
      [0.4421, 0.4380, 0.4339,  ..., 0.3843, 0.3843, 0.3719]]],


    [[[0.1240, 0.0992, 0.1240,  ..., 0.1983, 0.1818, 0.1653],
      [0.1157, 0.1033, 0.1281,  ..., 0.1901, 0.1570, 0.1488],
      [0.1074, 0.1074, 0.1240,  ..., 0.1694, 0.1488, 0.1364],
      ...,
      [0.9008, 0.6901, 0.2810,  ..., 0.5620, 0.6488, 0.4959],
      [0.9132, 0.6983, 0.3099,  ..., 0.5455, 0.6198, 0.5744],
      [0.9091, 0.7562, 0.2851,  ..., 0.4793, 0.6198, 0.6157]]]

What i meant by the third question is that usually when we download dataset from torchvision there is a parameter called transform where we can pass in ‘transformation’ variable which includes the mean and std() values, as a result it goes through the whole dataset and normalizes it. My question is after i calculate mean and std() for my custom dataset how do i apply it in order to normalize my data so that it goes through for all the batches and normalizes it ?

transformation = transforms.Compose([transforms.ToTensor(),
                               transforms.Normalize((0.1307,),(0.3081,))])
train_dataset = torchvision.datasets.MNIST('./data',train=True,transform= transformation, download= True)
4

I was able to apply transformation on a downloaded images(custom dataset) , But even though i applied the mean and standard deviation i am still getting test accuracy 0. While training the loss is decreasing, and after about 20 epoch it is 0. So i dont understand why i am getting 0% accuracy for the test set. Is my transformation code is correct? i had to define the downloaded data twice (‘data’) becuase if i dont define it in the __get__item it will throw an error. If i don’t define it in init i will not be able to add the channel dimension by unsqueezing, since the dataset comes as 4006464 format. Pls have a look at the code and tell me if u find any bug.

transformations = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5444,),(0.1581,))])
class dataset_transformed():

def __init__(self,transform=None):
    data = np.load("./Downloads/11928_16420_bundle_archive/olivetti_faces.npy")
    target = np.load("./Downloads/11928_16420_bundle_archive/olivetti_faces_target.npy")
    self.x = torch.from_numpy(data)
   
   
    self.x = self.x.unsqueeze(1)
   
    self.y = torch.from_numpy(target)
    self.y = self.y.type(torch.LongTensor)
   
    self.transform = transform
   
def __getitem__(self,index):
    data = np.load("./Downloads/11928_16420_bundle_archive/olivetti_faces.npy")
   
   
    if self.transform is not None:
        
        data = self.transform(data)
        
       
        
   
    return self.x[index],self.y[index]

def __len__(self):
    
    return self.n_samples

 dataset = dataset_transformed(transformations) #calling the class
5

The code might be correct, but since I don’t have the data you could check some samples and make sure they are properly normalized.

A 0% accuracy during testing might not necessarily be created by a wrong transformation.
Since you would expect to get at least the random accuracy, I would suggest to print some predictions as well as the targets and check, if the accuracy is really 0 or if you are calculating the accuracy in a wrong way.

6

Please i need a guide on how to calculate the mean and std of a dataset with images of different sizes, thanks

7 
train_loader = DataLoader(dataset=train_set, batch_size=64, shuffle=True)

def get_mean_std(loader):
    # var[X] = E[X**2] - E[X]**2
    channels_sum, channels_sqrd_sum, num_batches = 0, 0, 0

    for data, _ in tqdm(loader):
        channels_sum += torch.mean(data, dim=[0, 2, 3])
        channels_sqrd_sum += torch.mean(data ** 2, dim=[0, 2, 3])
        num_batches += 1

    mean = channels_sum / num_batches
    std = (channels_sqrd_sum / num_batches - mean ** 2) ** 0.5

    return mean, std


mean, std = get_mean_std(train_loader)
print(mean)
print(std)
5 Likes
8

Just my opinion:

Use torch.transforms to resize to same size for all images (just like your training), then find the mean and std.