DataLoader Filenames in each Batch

Yufeng_Ma · June 22, 2017, 7:49am

Hi, everyone,

I’m pretty new to PyTorch and am working with DataLoader to wrap my own image dataset. Suppose I’ve trained a binary image classifier, now I want to use this model to pick out the images that this model has misclassified. How can I get this done?

I’m thinking about working with the DataLoader class, but what I can get is only the transformed tensor batches from sampled images. I know which tensor in each batch is misclassified, but it’s hard for me to get access to their filenames. Anyone have similar concerns and can provide any workaround? Any suggestion is very welcome.

I really appreciate your help. Thanks.

smth · June 22, 2017, 2:53pm

you can write a custom Dataset that not only returns the images but also their ids / paths.
For example:

github.com

pytorch/vision/blob/master/torchvision/datasets/folder.py#L122




def __getitem__(self, index):
    """
    Args:
        index (int): Index


    Returns:
        tuple: (image, target) where target is class_index of the target class.
    """
    path, target = self.imgs[index]
    img = self.loader(path)
    if self.transform is not None:
        img = self.transform(img)
    if self.target_transform is not None:
        target = self.target_transform(target)


    return img, target


def __len__(self):
    return len(self.imgs)

can be

return image, target, path

or

return image, target, index

Yufeng_Ma · June 23, 2017, 2:04am

Great. Thanks a lot.

ajong · June 27, 2018, 5:11pm

For anyone who stumbles upon this later, I made a convenient little gist for this:

gist.github.com

https://gist.github.com/andrewjong/6b02ff237533b3b2c554701fb53d5c4d

pytorch_image_folder_with_file_paths.py

import torch
from torchvision import datasets

class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image paths. Extends
    torchvision.datasets.ImageFolder
    """

    # override the __getitem__ method that dataloader calls
    def __getitem__(self, index):

This file has been truncated. show original

vishalthengane · March 11, 2019, 10:43am

it work, Thank you!