After looking at the implementation and searching for similar issues, it seems nn.DataParallel is not supported for detection models, as described here.
We usually advise against using nn.DataParallel and to use nn.DistributedDataParallel instead, which seems to work out of the box for these models (and is also suggested by @fmassa in the linked issue).
Could you try it and let me know, if you encounter any issues using DDP?