Broken pipe for data_loading_tutorial on windows10

1

Hi everyone,
I get the broken pipe error while running the data_loading_tutorial code on my Windows10 machine but fine on my Linux machine.
the code is from the pytorch website:
https://pytorch.org/tutorials/beginner/data_loading_tutorial.html

I googled and find relevant problem for cifar10_tutorial code on Windows10. someone recommended wrapping the code with
if name == ‘main’:
But this solution does not work with this code. when set the num_workers = 0 then it works fine.
data_loader = DataLoader(face_dataset, batch_size=4,
shuffle=True, num_workers=1)
Does anyone have a similar problem?

2

From the error description it still looks like the Windows fork/spawn issue.
Did you make sure to wrap your complete code into a function and call it in the guard:

def main():
    # your complete code

if __name__=='__main__':
    main()
2 Likes
3

Thanks for your suggestion. I tried but now I have a new error:
Can’t pickle local object ‘main..FaceLandmarksDataset’
Still when I set num_workers=0, this error will be gone? Any solutions? I use Spyder. and Python 3.6
My code is as follows:

# -*- coding: utf-8 -*-

from __future__ import print_function, division
import os
import torch
import pandas as pd
from skimage import io, transform
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils

# Ignore warnings
import warnings
warnings.filterwarnings("ignore")

def main():    
    plt.ion()   # interactive mode

    class FaceLandmarksDataset(Dataset):
        """Face Landmarks dataset."""
    
        def __init__(self, csv_file, root_dir, transform=None):
            """
            Args:
                csv_file (string): Path to the csv file with annotations.
                root_dir (string): Directory with all the images.
                transform (callable, optional): Optional transform to be applied
                    on a sample.
            """
            self.landmarks_frame = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform
    
        def __len__(self):
            return len(self.landmarks_frame)
    
        def __getitem__(self, idx):
            img_name = os.path.join(self.root_dir,
                                    self.landmarks_frame.iloc[idx, 0])
            image = io.imread(img_name)
            landmarks = self.landmarks_frame.iloc[idx, 1:].as_matrix()
            landmarks = landmarks.astype('float').reshape(-1, 2)
            sample = {'image': image, 'landmarks': landmarks}
    
            if self.transform:
                sample = self.transform(sample)
    
            return sample
    
    
    
    transformed_dataset = FaceLandmarksDataset(csv_file='faces/face_landmarks.csv',
                                               root_dir='faces/')
    
    dataloader = DataLoader(transformed_dataset, batch_size=4,
                            shuffle=True, num_workers=4)
    enumerate(dataloader)
    
if __name__=='__main__':
    main()
4

Could you put the class definition outside of main.
Sorry for being unclear about it, but the functions and classes can be defined at the “global” level.
Just put your training code etc. into main.

5

It worked ! Thank you very much. I really appreciate your help. I am new to python. I used to play with C and matlab.