Hello I was training my fasterrcnn model a couldn’t get past a training loss of .19. At this time my number of works for my data loader was zero. I decided to try 4 for my number of works and suddenly my training loss got down to .03! How did this happen? I thought a number of workers was only for contributing to the speed of the model.
Yes, your assumption is correct that the number of workers should not change any hyperparameters etc. However, it would be interesting to check if you are accidentally reusing the same random transformations e.g. by not seeding the code properly.
See this section of the docs.
For transformations are you talking about image augmentation? I don’t have anything for transforms other than resizing the image from 416 to 1024. Are there any other links about this seeding and why it is important?
If no random transformations are used, the seeds shouldn’t matter as the samples should be transformed to the same outputs.
Since no randomness should be introduced, you could compare both DataLoaders and see where the samples would differ.
Iterate both loaders, store the samples in corresponding lists, and compare the samples for differences afterwards. You might need to disable shuffling (or store the index and sort it afterwards) and depending on the size of the dataset you might want to check subset(s).