How to normalize a custom dataset

vision
1

I followed the tutorial on the normalization part and used torchvision.transform([0.5],[0,5]) to normalize the input. My data class is just simply 2d array (like a grayscale bitmap, which already save the value of each pixel , thus I only used one channel [0.5]) stored as .dat file. However, I find the code actually doesn’t take effect. The input data is not transformed. Here is the what I tried:

class DatDataSet(Dataset):
    def __init__(self,root_dir,transform=None):
        self.root_dir=root_dir
        self.transform=transform
        self.label=os.listdir(data_dir)
        self.filepath=[os.path.join(data_dir,x) for x in self.label]
        fullpath=[list(map(lambda i: x+'\\'+i,os.listdir(x))) for x in self.filepath]
        self.datpath=reduce(operator.add,fullpath)
        self.num=[len(x) for x in fullpath]
        self.labellist=np.concatenate([counter*np.ones(x) for counter,x in enumerate(self.num)])
        self.transform=transform
        
    def __len__(self):
        return np.sum(self.num)
        
    def __getitem__(self,idx):
        img=pd.read_table(self.datpath[idx],header=None,sep='\s+')
        sample={'data':img.values,'label':int(self.labellist[idx])}
        return sample

transform=transforms.Compose([transforms.Normalize([0.5],[0.5])])

In order to see the normalization, I defined two data set, one with transformation dat_dataset = DatDataSet(root_dir=data_dir,transform=transform) and another without transformation dat_dataset2 = DatDataSet(root_dir=data_dir,transform=None)

Strangely, when I take the same index of data from dat_dataset and dat_dataset2, I found the values are the same.

Is it true that the torchvision.transform([0.5],[0,5]) can only transform the images instead of any custom dataset? Or what is the proper way to normalize?

Any comments and idea are highly appreciated. Thank you!

2

transforms.Normalize([0.5], [0.5]), as stated in the documentation, can be applied to Tensors only! Therefore you need to add another transform in your transforms.Compose() argument list: the ToTensor transform.

It would look like this:

transform = transforms.Compose([transforms.ToTensor,
                                transforms.Normalize([0.5], [0.5])])

Your code should have failed, because applying Normalize() on images does not work, but it hasn’t, since you never actually called the self.transform function on your image.

For this, you need to write the following, after loading the image (assuming that your pandas table contains the complete filepath for the images) in the __getitem__() function:

from PIL import Image
img_pil = Image.open(img)
img_normalized = self.transform(img_pil)
3

Thanks for your reply! May I ask why should I use Image.open ? Because the img imported by pandas is DataFrame. What does Image.open do here?

4

Sorry about that, I infered that you worked with PIL Images, which is the format recognized by torchvision transforms! However, I do not know the way you store the images in your dataset, could you provide more information on your dataset?

5

My raw data stored on the harddisk is tabular dat file. For example

1    2
3    4

Then I import the data using pandas, thus, img is the panda dataframe. img.values convert this dataframe to normal array:

array([[8.8635123e-01, 5.0047553e-01, 6.9067048e-01, ..., 6.9067048e-01,
         5.0047553e-01, 8.8635123e-01],
        [3.1736772e-01, 4.9979461e-01, 4.5827276e-01, ..., 4.5827276e-01,
         4.9979461e-01, 3.1736772e-01],
        [3.1614803e-01, 4.9785078e-01, 4.5332954e-01, ..., 4.5332954e-01,
         4.9785078e-01, 3.1614803e-01],
        ...,
        [6.4057958e-18, 6.8499766e-18, 1.3891328e-17, ..., 1.1393926e-17,
         2.4889349e-17, 7.0346043e-18],
        [1.1750645e-18, 5.4330497e-18, 2.7927236e-18, ..., 7.4773645e-18,
         1.7836283e-17, 5.2071868e-18],
        [1.0609590e-18, 1.1636564e-18, 2.5262159e-18, ..., 4.1869932e-18,
         1.3754568e-17, 4.4672030e-18]])
6

Great, then instead of using Image.open(), which takes a filename as argument to open an image file, you can use Image.fromarray(array), and keep the rest as is!

7

Maybe I miss something…
I got this error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-89-38c16a257125> in <module>
----> 1 temp=dat_dataset[1];

<ipython-input-86-8b1954b7a19e> in __getitem__(self, idx)
     18         img=img.values;
     19         img_pil = Image.fromarray(img)
---> 20         img = self.transform(img_pil)
     21         sample={'data':img.values,'label':int(self.labellist[idx])}
     22         return sample

~\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py in __call__(self, img)
     47     def __call__(self, img):
     48         for t in self.transforms:
---> 49             img = t(img)
     50         return img
     51 

TypeError: object() takes no parameters

when I try to run temp=dat_dataset[1]; The complete code is,

class class DatDataSet(Dataset):
....
def __getitem__(self,idx):
        img=pd.read_table(self.datpath[idx],header=None,sep='\s+')
        img=img.values;
        img_pil = Image.fromarray(img)
        img = self.transform(img_pil)
        sample={'data':img,'label':int(self.labellist[idx])}
        return sample

transform=transforms.Compose([transforms.ToTensor,transforms.Normalize([0.0144],[0.0608])])

dat_dataset = DatDataSet(root_dir=data_dir,transform=transform)

Then this is the line where error pops: temp=dat_dataset[1];

8

Oh… I find why…

It must be transforms.ToTensor(), right? I just copied your previous code and there is no parentheses.

9

Yes, my bad, I was misled by the documentation, where the entry is called torchvision.transforms.ToTensor without the parentheses like the other transforms… It should be with parentheses!

10

Thank you! By the way, I found that if I do not intend to normalize the data by calling dat_dataset2 = DatDataSet(root_dir=data_dir) , i.e. let transform=None. It will say:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-100-7fb86a0566ae> in <module>
----> 1 tempp=dat_dataset2[1];

<ipython-input-91-584bf0a783d6> in __getitem__(self, idx)
     18         img=img.values;
     19         img_pil = Image.fromarray(img)
---> 20         img = self.transform(img_pil)
     21         sample={'data':img,'label':int(self.labellist[idx])}
     22         return sample

TypeError: 'NoneType' object is not callable

So can I impletement ‘no normalization’ by not calling transform=transform?

11

Right, something else that I have overlooked!

In that case, you need to have two different composed transforms, that you select accordingly when you create the datasets:

transform_valid = transforms.Compose([transforms.ToTensor()])
transform_train = transforms.Compose([transforms.ToTensor(), transforms.Normalize(...)])

In case you might want the images to stay images, and not tensors, you can also set transform=None when you call your dataset, but then you need something like this:

class class DatDataSet(Dataset):
....
def __getitem__(self,idx):
    img=pd.read_table(self.datpath[idx],header=None,sep='\s+')
    img=img.values;
    img_pil = Image.fromarray(img)
    if self.transform is None:
        img = img_pil
    else:
        img = self.transform(img_pil)
    sample={'data':img,'label':int(self.labellist[idx])}
    return sample

However, this will output PIL.Image objects.

12

Yeah, I have similar idea. So I just simply if self.transform is not None: will do.

1 Like