Out of memory error during evaluation but training works fine!

5

The volatile flag is deprecated. In the latest stable release (0.4.0) you should use a context manager:

with torch.no_grad():
    # Your eval code

Have a look at the website for install instructions.
You can find the migration guide here.

25 Likes
6

hi. ptrblck. thank you for your help. it works now.

7

Hit the same problem, same solution worked, pytorch 4.0.1. Seems like there might be something weird going on with the eval mode memory management.

1 Like
8

A relevant clear-cut answer on ‘model.eval()’ vs ‘with torch.no_grad()’ from @albanD:

6 Likes
9

Thanks Arul, That’s helpful…

Still doesn’t explain why eval mode appears to use more memory than training mode though. Theoretically it would just use the same.

5 Likes
10

I am a novice. Have the same problem but during the inference, I have never met ‘out of memory’ error without using the torch.no_grad() or volatile=True before. But at this time it seems not to work without using torch.no_grad(). pytorch 3.0.0.

11

+1, why use more memory than training?

1 Like
12

You might run out of memory if you still hold references to some tensors from your training iteration.
Since Python uses function scoping, these variables are still kept alive, which might result in your OOM issue. To avoid this, you could wrap your training and validation code in separate functions. Have a look at this post for more information.

2 Likes
13

@ptrblck , did you mean this post?

14

Yes, @colesbury explains, why the memory usage might grow if some tensors weren’t deleted using function scoping.

1 Like
15

What is the effect if you forget torch.no_grad() besides the increased memory. Will you accumulate gradients in the validation block?

16

The computation graph will be created and intermediate tensors are stored.
If you don’t call backward (which wouldn’t even be possible in a torch.no_grad() block), nothing else will change.

17

Well you would call backward in the training portion. So would you then update the net with the grads tracked in the validation portion as well as those in the training portion? Assuming that torch.no_grad() was forgotten in validation.

18

During training a new computation graph would usually be created, as long as you don’t pass e.g. the output of your validation phase as the new input to the model during training.

model = models.resnet18(pretrained=True)

# Pseudo validation phase
x1 = torch.randn(1, 3, 224, 224)
out = model(x1)

# Pseudo training phase
x1 = torch.ones(1, 3, 224, 224)
out = model(x1)
out.mean().backward()

In this code snippet you have “forgotten” to use torch.no_grad() during the validation phase.
However, since out is not used, it won’t have any effect on the gradients, but will just use unnecessary memory.

19

Ok cool, what about if it’s set up this way.

crit = nn.SomeLoss()
optim = optim.SGD()
net = models.resnet18()

for e in range(num_epochs):
    
    # training
    pred = net(some_data)
    optim.zero_grad()
    loss = crit(pred, target)
    loss.backward() 
    optim.step()

    # validation
    valid_pred = net(some_validation_data)
    loss = crit(valid_pred, valid_target)

Would zero_grad take care of that?

20

As long as you don’t calculate gradients via a backward call, no gradients will be accumulated.

21

Awesome, thanks for taking the time to answer my questions.

23

Thanks for your comment, was helpful for me too! :slight_smile:

1 Like
24

It actually works. I’ve trained torchvision’s DensNet161 with 1 GPU. The model was trained well in training stage, but out of memory in eval. stage. Inference of model was performed without ‘torch.no_grad()’. Now there’s no out of memory. Thanks.

25

You’ve saved my homework and life. There’s full of tears of gratitude in my eyes. QAQ