Are samples with no positive ignored in training Faster R-CNN?

I’m training Faster R-CNN on underwater fish detection. I have samples from rather monotonous scenes (only fishes in water). This leads the model to recognize every region proposal as positive in practice (any thing appears will be classified as fish).
I want to add some samples from other scenes (but with no positive) to have the model learn more features. Yet if samples with no positive are ignored in training, this attempt would be useless.