GPU utilisation becomes 0% periodically

vision
1

I am trying to finetune Densenet-121.

My dataloader looks like this


class gamma_corr:
    
    def __init__(self,gamma):
        self.gamma = gamma
        
    def __call__(self,x):
        return transforms.functional.adjust_gamma(x,self.gamma)

class resize:
    
    def __init__(self,factor):
        self.factor = factor
        
    def __call__(self,img):
        h,w = img.size
        return img.resize([round(h*self.factor),round(w*self.factor)],PIL.Image.BICUBIC)
   

resize = transforms.RandomChoice([resize(0.5),resize(0.8),resize(1.5),resize(2.0)])
gamma = transforms.RandomChoice([gamma_corr(0.8),gamma_corr(1.2)])
alt = transforms.RandomOrder([resize,gamma])

img = transforms.RandomChoice([alt,lambda x: x])

transformations = transforms.Compose([transforms.RandomCrop(960,960),img,transforms.RandomCrop(256,256),
                                      transforms.ToTensor(),transforms.Normalize([0.5,0.5,0,5],[0.5,0.5,0.5])])


dataset = datasets.ImageFolder('./train/',transform=transformations)
loader = DataLoader(dataset,batch_size=32,shuffle=True,num_workers=8,pin_memory=True,drop_last=True)

I have an 8 core CPU, the CPU utilization is always 100%
I guess the GPU consumes data a faster rate than the CPU can produce and thus has to wait.

How can I improve it ?.

2

It looks like you have a lot of transformations for your dataset, so that’s probably the reason for the starving of the GPU.

How are you loading your data? Do you have image files? Are they stored locally on an SSD?
If you are loading the image files in your Dataset, are you using Pillow-SIMD?
From the docs:

a much faster drop-in replacement for Pillow with SIMD. If installed will be used as the default.

3

Yes, they are image files and are stored locally on an SSD.

I did install pillow-SMID using pip but that isn’t helping much.

4

Did you find the bottleneck?

Are the transformations the crucial (slowest) part or is it the loading of the files itself? How large are your files? Did you try to create a hdf5 file of your dataset?

5

The problem persists even when no transformations are applied. So the bottleneck must be the loading itself.
Image files are in order of ~5MB.

I don’t know about this hdf5 thing. Can you please share a good resource about it.
Thanks for the help.

6

Have a look at the introductory python examples.

Note: you need to have h5py installed