Bug: MaxPool3d causes GPU memory leaking

When I try to train my model which contains MaxPool3d , It always end up with ‘out of memory’ error.
my environment info is here:

Ubuntu 16.04
PyTorch version: 0.4.0a0+da6c3c9
installed PyTorch from source
python version is 3.6.2
CUDA/cuDNN version: 9.1/7.1.2
GCC version (if compiling from source):GCC 5.4
Build command you used (if compiling from source): as default
I can reproduce this bug by the following script:

import torch
import torch.nn as nn
from torch import optim
from torch.nn.functional import smooth_l1_loss
model = nn.Sequential(
    nn.MaxPool3d(kernel_size=3, stride=2, padding=1),
#optimizer = optim.SGD((x for x in model.parameters() if x.requires_grad is True), lr=1e-3, momentum=0.9,nesterov=True)
crit = smooth_l1_loss
count = 0
while True:
#    optimizer.zero_grad()
    input = torch.rand(30,1,200,200,200).cuda()
    loc_output= model(input)
    loc_outpus = loc_output
    if type(loc_output) == tuple:
        loc_output = loc_output[0]
    targets = torch.rand(loc_output.size()).cuda()
    loss = crit(loc_output,targets)
#    optimizer.step()
    del loss,loc_output,targets,input
    count += 1


Your code sample does have a high memory usage ~5GB but it does not leak memory on my setup. I have a stable memory usage of 4674MB after running 20 iterations.

Removing del loss,loc_output,targets,input and torch.cuda.empty_cache() did not increase the peak memory usage and kept a constant memory usage at the value above.

I’m using pytorch from master (0.4.0a0+4afd62d) with python 2.7 and same cuda/cudnn as you.

Thank you very much for your reply. This bug is also working well on the early commit node. However, after a recent update, this bug has appeared. We have been positioning for a long time only to find out in maxpool3d.
We submitted it on ‘issues’ and @zou3519 reproduces the bug.

Perfect !
I should keep my old master until this is fixed then :smiley:

1 Like