How do i write a custom Data Loader?

Hi, I’m new to pytorch. Is there a guideline to write your own Data Loader?

Thank you.

Hi,

Here is the official custom data loading tutorial. Also, this question has been answered for many different situations in this forum.

Bests

Sorry, still did not understand data loader.
Is this part the Custom Data Loader in the tutorial?
“”“Face Landmarks dataset.”""

def __init__(self, csv_file, root_dir, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.landmarks_frame = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = transform

def __len__(self):
    return len(self.landmarks_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_name = os.path.join(self.root_dir,
                            self.landmarks_frame.iloc[idx, 0])
    image = io.imread(img_name)
    landmarks = self.landmarks_frame.iloc[idx, 1:]
    landmarks = np.array([landmarks])
    landmarks = landmarks.astype('float').reshape(-1, 2)
    sample = {'image': image, 'landmarks': landmarks}

    if self.transform:
        sample = self.transform(sample)

    return sample

Actually, this is a custom Dataset which can be used as an input to a DataLoader. If you are looking for custom DataLoader, usually the only parameter you can modify is collate_fn which corresponds to batching.

What if I want to add functionality to data loader, I plan to add a function in data loader that send a request to a CPU cloud via MPI? Is it possible? Thankyou.

Oh, I have never worked with remote servers and clusters so I cannot answer this question, sorry.
I think if you create another topic and add this issue on the title so people can interact better. I was only thinking of batching and loading data in different approaches.

Sorry again, good luck

No worries.Thanks for the help.

Hey,
If you want to write a custom data loader you can check out my article : https://towardsdatascience.com/the-pytorch-training-loop-3c645c56665a

However, I would suggest adding the functionality to a custom dataset instead of a data loader.