Dataloader map-style dataset, how does it work?

1

I’m having trouble trying to fine-tune a model.
My main problem is that I cannot verify if the data is being correctly interpreted by the code.
The model will not advance and have a significant result, so I suspect the labels are messed up.

To set the data for training and validation I used the following code found in the Pytorch documentation.(Finetuning Torchvision Models — PyTorch Tutorials 2.1.1+cu121 documentation)

Create training and validation datasets

image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms) for x in [‘train’, ‘val’]}

Create training and validation dataloaders

dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets, batch_size=batch_size, shuffle=True, num_workers=0) for x in [‘train’, ‘val’]}

My data is organized in folders in “./database/train” and “./database/val”.
I’m treating different shoe models in different folders that I’d like to classify.

example of folders: “./database/train/ADIDAS - Yeezy 500 Bone White” and “./database/train/NIKE - OFF WHITE - Air Max 90 Desert Ore”

Where I’ve inserted multiple images of a given category (which is the name of the model).

So my main question is: when I use the torch.utils.data.Dataloader(), is it considering the name of the folder as the label and treating it automatically what is inside the folder? That’s what I thought it would do, given the map-style dataset functionality (torch.utils.data — PyTorch 2.1 documentation)

Thank you!!

2

The DataLoader does not interpret or create the targets, but creates batches of samples using the Dataset.
ImageFolder, which you have used as the dataset, will sort the folders and assign a class label to each folder.
You could check some samples by trying to visualize a few random indices:

x, y = image_datasets['train'][0]
plt.imshow(x.permute(1, 2, 0).numpy())
print(y)
print(image_datasets['train'].class_to_index)

Note that the visualization might look distorted, if you’ve normalized the data, so you could either remove the normalization for the sake of debugging or “unnormalize” the data.

1 Like
3

Thank you!! That was really helpful, so if I just separate the folders of different classes with different names this should take care of it for me?
The reason I asked it is because I though the Data loader in Pytorch was messing up the labels.
I’m fine tuning vgg11, resnet and other models in order to classify 77 shoe classes where I have about 100 to 150 samples of each class (about 12k images in total), about 70% for training and 30% for testing, the results wont get any better than 1/77=1.3% and I find it really awkward, even though I know I don’t have a good data-set.
Do you think that’s normal?
Thank you again and again, I really appreciate your help !!

4

Yes, this should be the case.

I would recommend to scale down the problem a bit and maybe just use two classes with 5 samples each. Once you’ve created this dummy dataset, try to overfit your model so that you reach ~100% accuracy by changing the learning rate etc.
If that’s not possible, there might be a bug in your code somewhere or the model architecture is “bad”, but I doubt the second point as VGG and ResNet models are heavily used.

1 Like