Dataloader gave no response

1

I am working on loading data in pytorch, I followed the official tutorial and wrote my own dataset, the code is shown as below and it worked fine:

import os
import numpy as np
import time
from torch.utils.data import Dataset, DataLoader


data_dir = 'D:/lily_data/data'
label_dir = 'D:/lily_data/label'
data_list = os.listdir(data_dir)
label_list = os.listdir(label_dir)

dataset_list = []
for i in range(len(data_list)-15):
    data_name = data_dir+'/'+data_list[i]
    label_name = label_dir+'/'+label_list[i]
    dataset_list.append([data_name, label_name])
     

class lily_dataset(Dataset):
    
    def __init__(self, data_list):
        self.data_list = data_list
        
    def __len__(self):
        return(len(self.data_list))
        
    def __getitem__(self, idx):
        data = np.load(self.data_list[idx][0])
        label = np.load(self.data_list[idx][1])
        return data, label
    
start = time.time()
dataset = lily_dataset(dataset_list)
#dataset = DataLoader(dataset, batch_size=2, shuffle=True, num_workers=2)
for idx, data in enumerate(dataset):
    print(idx, data[0].shape, data[1].shape, time.time()-start)
    start = time.time()

The result of this code is shown like this:

0 (512, 512, 800) (512, 512, 1) 0.8958175182342529
1 (512, 512, 800) (512, 512, 1) 1.065506935119629
2 (512, 512, 800) (512, 512, 1) 1.1634063720703125
3 (512, 512, 800) (512, 512, 1) 1.31596040725708
4 (512, 512, 800) (512, 512, 1) 14.15942931175232
5 (512, 512, 800) (512, 512, 1) 14.121654033660889

But when I uncomment this line

dataset = DataLoader(dataset, batch_size=2, shuffle=True, num_workers=2)

and run the code, it just give no response. I run this script in spyder, it seems to be running but yield no result or any response.
I think I just followed the tutorial and cannot figure out what’s wrong. Any help or idea would be appreciated.

2

If you are using Windows, you should guard all of your code with:

def main():
    # Your code

if __name__=='__main__':
    main()

Since Windows uses spawn instead of fork to create processes, your code will be executed multiple times.
Have a look at the Windows FAQ.

1 Like
3

Really thank you for your kind help, I tried and solved this problem.