High GPU Memory Demand for pytorch?

vision
22

You probably should create a new thread with your issue, this thread is 2 years old.

Also how do you check memory usage. As far as I know, once you have allocated GPU memory, PyTorch keeps it even when unused to quickly reuse it later without having to request it from Cuda, see Reserving gpu memory? so you can’t simply use nvidia-smi.

23

I have to test the volatile option… Yes I checked that on model training nvidia-smi reports same gpu usage that when evaluating (same batch and so on)

24

There is no volatile option anymore, you only need to use requires_grad = False to make sure a graph is not built.

Also if you want to return memory to the GPU just del your_tensor but be aware that GPU memory allocation/deallocation is very costly hence why all frameworks prefer to keep memory around even unused: https://www.cs.virginia.edu/~mwb7w/cuda_support/memory_management_overhead.html

25

So… it’s normal that using model.eval() and run the model forward inside a

with torch.no_grad():
    model(data_from_datalaoder)

uses the same memory (nvidia-smi) as the train phase? If I increase the batch of eval only model runs out of memory easily