Extremely slow data-loading on first epoch

kishwarshafin · July 19, 2018, 8:03pm

My dataloader is extremely slow on first epoch. I have tried to go through all the solutions and threads posted online and almost nothing helps. These are the things I did:

Tested on SSD which also has an NVMe support. The data transfer rate is extremely fast.
Turned off pin_memory, doesn’t help.
Moved from reading one HDF5 file to individual hdf5 files.
Removed zipping hdf5 data.

My model is very small for practical purposes and the batch size is also small. But as the amount of data increases the dataloader gets slower and slower on the first epoch. Is there any solution to this?

Dataloader link:

github.com

CGL-Deeplearning/FRIDAY/blob/dev_v0.1_w10_resnet/modules/core/dataloader.py

import os
import numpy as np
import pandas as pd
from torch.utils.data import Dataset
import h5py
import torch
import pickle


class SequenceDataset(Dataset):
    """
    Arguments:
        A CSV file path
    """

    def __init__(self, csv_path, transform=None):
        data_frame = pd.read_csv(csv_path, header=None, dtype=str)
        # assert data_frame[0].apply(lambda x: os.path.isfile(x.split(' ')[0])).all(), \
        #     "Some images referenced in the CSV file were not found"
        self.transform = transform

This file has been truncated. show original

Train script:

github.com

CGL-Deeplearning/FRIDAY/blob/dev_v0.1_w10_resnet/train.py

import argparse
import os
import sys
import time
import random

import torch
import torch.nn.parallel
import torchnet.meter as meter
import torch.nn as nn
from tqdm import tqdm
from torch.utils.data import DataLoader
from torchvision import transforms
from modules.core.dataloader import SequenceDataset
from modules.models.ModelHandler import ModelHandler
from modules.handlers.TextColor import TextColor
"""
Train a model and save the model that performs best.

Input:

This file has been truncated. show original

aplassard · July 19, 2018, 8:06pm

What is your environment? Are you using a GPU?

kishwarshafin · July 19, 2018, 8:14pm

Yes,
pytorch-0.4.0
cuda 9.2 Driver 396.26
8-gpus- 1080ti

Ubuntu 16.04, python3.5 (if you meant this!)

aplassard · July 20, 2018, 11:05am

I’m curious, if you run the model on the CPU, does it have the same issue? Also, do you know which line(s) it is hanging on longest?

kishwarshafin · July 20, 2018, 9:43pm

I think I have a temporary solution for now. I have split my dataloader into train and test. In train I don’t send the string tuples I was sending before. I only send the image and it’s associated label during training. It kind of solves the issue temporarily, I’ll have to run big tests again. I’ll post if the issue becomes an issue again.

Cheers.