Calling backward() Twice Lead to GPU Memory Leak?

Hi,

When I train the network, GPU memory will grow up continually until out of memory. Is it memory leak?

The code is
https://paste.ubuntu.com/26269199/

This is a simple fully connected network to deal with a regression problem. I assume that the key of the problem is I use backward() to get gradients of output, then I use backward() again to optimize weights.

I try some suggestions:

  1. delete variables
  2. add torch.backends.cudnn.enabled = False
  3. add gc.collect()
    … but the problem is still unresolved.

Please give me any hints or suggestions. Thanks!

—————
Vesions
pytorch 0.4.0a0+1b608ee
python 3.5.3
ubuntu 16.04

Hi,

It depends on the type of data that you are using, but try reducing the batchSize

Thanks for your advice!

Here’s the dataset that I’m using:

# fake data to show the problem
x = torch.cuda.FloatTensor(10000,3)
x = torch.randn(10000,3)
y = torch.cuda.FloatTensor(10000,3)
y = torch.randn(10000,3)

train = data_utils.TensorDataset(x, y)
train_loader = data_utils.DataLoader(train, batch_size=batchSize, shuffle=False)

I try to set the batchSize to 1, but the GPU memory is still keeping increasing during training.