Hello,
I am doing a grid search over many different hyper parameters. I wrote a script for this task that is generating all combinations of hyperparameters, then forks one thread for each GPU (I have 4 GPUs in the machine, so I use 4 threads) and then each thread trains a model. There is a queue with all hyperparameter configurations and each thread gets its current configuration from this list.
Each thread is doing these steps:
- Read training and validation samples from h5 file
- Initialize DataLoaders that also do some transformations (RandomFlip, Normalization, etc.)
- Train for N epochs and validate after each epoch
- Save results to a file
However, there is some strange bug that causes my script to freeze randomly. Sometimes in the first epoch, sometimes later and always on different batches within the epoch. nvidia-smi is showing that the GPU of the thread that freezes still uses memory but has 0% load.
±----------------------------------------------------------------------------+
| NVIDIA-SMI 390.67 Driver Version: 390.67 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX TIT… Off | 00000000:02:00.0 Off | N/A |
| 22% 26C P8 15W / 250W | 547MiB / 12212MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 1 GeForce GTX TIT… Off | 00000000:04:00.0 Off | N/A |
| 22% 27C P8 14W / 250W | 582MiB / 12212MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 2 GeForce GTX TIT… Off | 00000000:83:00.0 Off | N/A |
| 22% 32C P8 14W / 250W | 563MiB / 12212MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 3 GeForce GTX TIT… Off | 00000000:84:00.0 Off | N/A |
| 31% 70C P2 122W / 250W | 3411MiB / 12212MiB | 60% Default |
±------------------------------±---------------------±---------------------+±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 19757 C python 526MiB |
| 1 19757 C python 561MiB |
| 2 19757 C python 542MiB |
| 3 19757 C python 3390MiB |
±----------------------------------------------------------------------------+
I have added many print statements to find the position where it freezes and I guess that the ToPILImage() transformation is the problem. The error does only occur when I use num_workers > 0 in my DataLoaders.
I have already seen a few bug reports that had a similar problem when using cv2 in their Datasets (that are used by DataLoader). But I am not using cv2, only torch, torchvision and (indirectly) PIL / pillow. Is there a known issue when using DataLoader within a thread? Is it not possible to run multiple training scripts that all use DataLoaders in parallel?
When I use num_workers = 0, it works, but then it is quite slow.
Here is some relevant code:
jitterTransform = transforms.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1)
normMean = tuple(np.array([126.55891472, 99.32252242, 97.26477874])/255.0)
normStd = tuple(np.array([65.72192038, 56.78839458, 59.06476169])/255.0)
normTransform = transforms.Normalize(normMean, normStd)
trainTransform = transforms.Compose([
transforms.ToPILImage(),
jitterTransform
])
testTransform = transforms.Compose([
transforms.ToPILImage(),
transforms.ToTensor(),
normTransform
])
with h5py.File(self.data, 'r') as f:
train_dataset = MyDataset(f['train'], transform=trainTransform, flip=True, trans=normTransform, gpu=self.gpu_id)
val_dataset = MyDataset(f['test'], transform=testTransform, gpu=self.gpu_id)
dataloader_params = {
'batch_size' : config['batch_size'],
'pin_memory' : True,
'num_workers' : 32 # High value => Freezes faster
}
train_loader = DataLoader(train_dataset, shuffle=True, **dataloader_params)
val_loader = DataLoader(val_dataset, shuffle=False, **dataloader_params)
and MyDataset:
from torch.utils.data import Dataset
class MyDataset(Dataset):
def __init__(self, f, transform=None, flip=False,trans=None):
self.images, self.coords = f['images'][:], f['coords'][:]
if transform is None:
self.images = np.transpose(self.images, (0, 3, 1, 2)) # swap from B x H x W x C to B x C x H x W
self.transform = transform
T=RandomHorizontalFlip(0.5,trans=trans)
self.maybe_flip = (lambda x: T(x)) if flip else (lambda x: x)
def __len__(self):
return len(self.images)
def __getitem__(self, idx):
image=self.images[idx]
landmark=self.coords[idx]
if self.transform:
print("before transform")
image = self.transform(image)
print("after transform")
sample = {'image': image, 'landmarks': landmark}
return self.maybe_flip(sample)
In the cases where it freezes, I see only the “before transform” output but not the “after transform” output.
Is there a known issue with race conditions / deadlocks / etc when using ToPILimage in multiple threads in parallel?
EDIT: As soon as 3 of my 4 GPU threads freeze, the last one continues running without any problem. This leads to the assumption that my code in general is not wrong, but there are some strange side effects when it is run in more then one thread.
EDIT 2: Even when I only use this transform and num_workers > 0, it still freezes:
transforms.ToTensor()
So it does not have something to do with PIL? I am completely confused now. It looks like there is some bug in the DataLoader itself when it is used in multiple threads… Even if every thread has its own DataLoader.
Moving this from vision to the uncategorized topic since I think PIL is not involved.