Passing negative samples data into FasterRCNN

Hello everyone
I am using FaterRCNN network and provide [0.0, 0.0, 0.0, 0.0] boxes with label=0, as a negative example.
I get the error ValueError: All bounding boxes should have positive height and width. Found invalid box [0.0, 0.0, 0.0, 0.0] for target at index 1.
Additional info:

@ptrblck tagging you, because you are some kind of guardian angel of this forum)