GPU memory usage growth slowly and result in OOM

I implemented a Network using ROI-pooling, But got a problem that the usage of GPU memory grows as training, after iteration to 50, OOM occurs.
I use resnet34 as the basic net, and insert ROI pooling layer between layer3 and layer4, with image size 3*1024*2048

at the first iteration, uses only less than 5GB memory, but with training going, usage grows

And how could I debug this issue?