I implemented a Network using ROI-pooling, But got a problem that the usage of GPU memory grows as training, after iteration to 50, OOM occurs.
I use resnet34
as the basic net, and insert ROI pooling
layer between layer3 and layer4, with image size 3*1024*2048
at the first iteration, uses only less than 5GB memory, but with training going, usage grows
And how could I debug this issue?