CUDA error when I call a preprocess function within a DataLoader


I have a network which is the combination of a CenterTrack network (GitHub - xingyizhou/CenterTrack: Simultaneous object detection and tracking using center points.) and a custom network that is fed with the embedding from the CenterTrack net. Original images goes through CenterTrack and I branch out a couple of heads before CenterTrack starts upsampling.

The non efficient implementation of this model works fine, meaning that I get the output I expect. The only problem is that the whole process takes too long.

The inefficiency is due to the fact that the only way to make CenterTrack net working is through a for loop that iterates through every single image of a batch, applying the Centertrack network and extracting the embeddings. Indeed Centertrack seems to work only on single images. This is what aurthors do in “”.

In order to get rid of the for loop, I tried to call the CenterTrack detector within getitem in the DataLoader. This is the most natural solution in my head. But such an implementation I get the following error:

““RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method””

I get this error both if I initialise the CenterTrack in the DataLoader itself and also if I initialize it outside the dataloader, passing the CenterTrack instance to teh DataLoader as a parameter.

Does anyone know why I get this error and how I can fix it?

Than you

The error is raised, since you are most likely using multiprocessing in the DataLoader by specifying num_workers>0, which would then try to re-initialize the CUDA context.
Did you try the suggested workaround using the 'spawn' start method?