How did you time your script?
Does your model run on CPU or GPU?
Could you post an executable script? We’ve had a similar issue here and realized the model just performed differently using random weights (i.e. creation of more detection candidates).