Hi!
So this is a minimal code which illustrates the issue:
This is the Dataset:
class IceShipDataset(Dataset):
BAND1='band_1'
BAND2='band_2'
IMAGE='image'
@staticmethod
def get_band_img(sample,band):
pic_size=75
img=np.array(sample[band])
img.resize(pic_size,pic_size)
return img
def __init__(self,data,transform=None):
self.data=data
self.transform=transform
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
sample=self.data[idx]
band1_img=IceShipDataset.get_band_img(sample,self.BAND1)
band2_img=IceShipDataset.get_band_img(sample,self.BAND2)
img=np.stack([band1_img,band2_img],2)
sample[self.IMAGE]=img
if self.transform is not None:
sample=self.transform(sample)
return sample
And this is the code which fails:
PLAY_BATCH_SIZE=4
#load data. There are 1604 examples.
with open('train.json','r') as f:
data=f.read()
data=json.loads(data)
ds=IceShipDataset(data)
playloader = torch.utils.data.DataLoader(ds, batch_size=PLAY_BATCH_SIZE,
shuffle=False, num_workers=4)
for i,data in enumerate(playloader):
print(i)
pass
It gives that weird open files error in the for loop…
And this is the state of my machine when getting that weird error:
yoni@yoni-Lenovo-Z710:~$ lsof | wc -l
89114
yoni@yoni-Lenovo-Z710:~$ cat /proc/sys/fs/file-max
791958
My torch version is 0.3.0.post4
If you want the json file, it is available at Kaggle (https://www.kaggle.com/c/statoil-iceberg-classifier-challenge)
What am I doing wrong here?! 