Object Detection

I am currently building pytorch version of YOLO… But I have some difficulty in building multi scale training. Do I have to build the various kind of networks for that feature for various sizes.