Problem with 'num_workers' in my data_loader

Hi, there I am trying to train a model and getting a strange error…

If I run the model on a small subsection of my data without the data loader argument ‘num_workers’ the model runs fine…

However, if I attempt to run the model on the whole dataset (or even a significant amount of the dataset) the model hits a bottleneck - still without ‘num_workers’.

I figured the simple solution to this error is to set ‘num_workers’ - I have tried a number of different workers up to 32. This is where I get my error as seen below…

In summary, if I want to train my model on the whole dataset I am stuck as the model either bottlenecks when ‘num_workers’ is removed from the data loader or I get this error if I set it to anything else.

I guess this is not the problem occurred by num_workers.
I don’t know how you sampled a subset but you’d better unite the format of your dataset.

Check the data type

1 Like