Extending torchvision for handling own datasets

ankghost0912 · May 19, 2017, 11:47am

Hi there,

I’m very new to PyTorch so please bear with me. I’ve been following and reading tutorials to get familiar with pytorch. The tutorials all use torchvision package which contains dataloaders for CIFAR-10/100, COCO etc. I wanted to know if torchvision’s functionality can be extended to any non-standard dataset that I may have.

If not, then can one write their own custom dataloaders and still use the transforms features defined in torchvision?

Thanks

chenyuntc · May 19, 2017, 12:28pm

Of course, as long as you write your own Dataset which is very easy to implement. Then you can utilize the speedup of multiprocessing by using Dataloader

You may refer to Imagefolder it’s a standard implementation of Dataset.

github.com

pytorch/vision/blob/master/torchvision/datasets/folder.py#L62-L91






def default_loader(path):
from torchvision import get_image_backend
if get_image_backend() == 'accimage':
    return accimage_loader(path)
else:
    return pil_loader(path)




class ImageFolder(data.Dataset):
"""A generic data loader where the images are arranged in this way: ::


    root/dog/xxx.png
    root/dog/xxy.png
    root/dog/xxz.png


    root/cat/123.png
    root/cat/nsdf3.png
    root/cat/asd932_.png

This file has been truncated. show original

ankghost0912 · May 19, 2017, 12:48pm

Thanks, just one more question - Does this class only support image data as of now or it can be used without any modifications in cases like text data for training RNNs.

chenyuntc · May 19, 2017, 1:11pm

It supports all kind datasets, and it could also be used to load raw text file.But you need to write your own loader(read file into memory) and transform(transform text data to tensor).
As for text datasets, try:

ankghost0912 · May 19, 2017, 1:14pm

Thanks for helping a beginner out!