Since I want to adopt multi-scale training for object detection, the image sizes would be changed per fixed frequency. When using dataparallel, the image sizes trained on each GPU can be synchronized.
To speed up the training phase, I want to use DistributedDataParallel, but I don’t know how to synchronize the input image size. Any suggestions, please?
Hi!
One way to do it is to resize your batches in a custom collate function that you send to your dataloader. This collate function would make sure that all the images within one batch have the same dimension.
I would try to get this working before you start with the DistributedDataParallel. Actually, depending on your multi-scale schedule (fixed-frequency), the DistributedDataParallel might not offer any difficulties once you got the collate function up and running.
Google + search at these forums on how to implement a custom collate function for the dataloader and give me a poke if you want to talk something over.
Good luck 
Edit: Since it’s object detection you also need to transform your bounding boxes. I recommend using imgaug for this, so much easier