# How to calculate the GPU memory that a model uses? **URL:** \ **Category:** Uncategorized\ **Created:** [July 26, 2022, 6:03am UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486 "2022-07-26T06:03:50Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![may1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/may1/32/45344_2.png) [@may1](https://discuss.pytorch.org/u/may1)\ **Post date:** [July 26, 2022, 6:03am UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/1 "2022-07-26T06:03:50Z")
I wanted to reduce the size of Pytorch models since it consumes a lot of GPU memory and I am not gonna train them again. First, I thought I could change them to TensorRT engine. and then I was curious how I can calculate the size of gpu memory that it uses. Pytorch model size can be calculated by ```auto torch.cuda.memory_allocated ``` or calculating using model.parameters() and model.buffers() I checked if the above results had same values and they had. But the size of TensorRT engine or other Scripted modules for GPU cannot be calculated by the above torch functions. So I thought I could check the gpu memory usage size with GPUtil library. However, the memory usage size that was calculated by GPUtil library (using nvidia-smi) was too different. For example, one model has 13MiB size but almost 2 GiB was allocated in GPU. The other model has 171MiB but also around 2GiB was allocated in GPU. I didn’t put other objects such as inputs in GPU. and Even after deleting the model, ```auto del model gpu = GPUtil.getGPUs()[0] memoryUsed = gpu.memoryUsed ``` memory was still not 0, while torch.cuda.memory\_allocated(0) shows 0. how do you calculate the GPU memory that a pytorch model uses? or how do you compare the GPU memory that a pytorch model uses and its script-mode uses? and if I understood right and used the right functions, why is the actual allocated memory that different from real torch tensor bytes? I knew it could be different because of using defined page sizes but I didn’t expect that it could be that much different (2GB difference). ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 26, 2022, 6:57am UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/2 "2022-07-26T06:57:17Z")
PyTorch will create the CUDA context in the first CUDA operation, which will load the driver, kernels (native from PyTorch as well as used libraries etc.) and will take some memory overhead depending on the device. PyTorch doesn’t report this memory which is why `torch.cuda.memory_allocated()` could return a 0 allocation. You would thus need to use `nvidia-smi` (or any other “global” reporting tool) to check the overall GPU memory usage. ---
**Author:** ![ndvbd](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ndvbd/32/56902_2.png) [@ndvbd](https://discuss.pytorch.org/u/ndvbd)\ **Post date:** [May 25, 2023, 1:04pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/3 "2023-05-25T13:04:24Z")
Is there a way to measure the peak gpu memory consumption between two points in time? For example, I am running a forward call on some network. Memory start at value x, then during the forward call goes to y, and then at the end back to x- I would like to discover the y value. ---
**Author:** ![WANGSSSSSSS](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wangsssssss/32/50338_2.png) [@WANGSSSSSSS](https://discuss.pytorch.org/u/WANGSSSSSSS)\ **Post date:** [May 25, 2023, 1:06pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/4 "2023-05-25T13:06:51Z")
`torch.cuda.max_memory_allocated(*device=None* )` is all you need! [torch.cuda.max\_memory\_allocated](https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html#torch.cuda.max_memory_allocated) ---
**Author:** ![ndvbd](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ndvbd/32/56902_2.png) [@ndvbd](https://discuss.pytorch.org/u/ndvbd)\ **Post date:** [May 25, 2023, 1:32pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/5 "2023-05-25T13:32:39Z")
Thank you. Strangely however, this does not yield 0: ```auto torch.cuda.reset_peak_memory_stats(device=None) print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory") ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 25, 2023, 4:45pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/6 "2023-05-25T16:45:44Z")
It works for me: ```python print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory") # gpu used 0 memory x = torch.randn(1024, device="cuda") y = torch.randn(1024, device="cuda") print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory") # gpu used 8192 memory # should return same memory usage as nothing was deleted torch.cuda.reset_peak_memory_stats(device=None) print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory") # gpu used 8192 memory # delete tensors and reduce peak memory usage del y torch.cuda.reset_peak_memory_stats(device=None) print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory") # gpu used 4096 memory del x torch.cuda.reset_peak_memory_stats(device=None) print(f"gpu used {torch.cuda.max_memory_allocated(device=None)} memory") # gpu used 0 memory ``` ---
**Author:** ![ndvbd](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ndvbd/32/56902_2.png) [@ndvbd](https://discuss.pytorch.org/u/ndvbd)\ **Post date:** [May 25, 2023, 7:56pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/7 "2023-05-25T19:56:38Z")
I’ve put it in a random place in my program, and it didn’t show 0. It should, right? There is occupied memory by the GPU, but if it’s right after the reset\_peak, I guess it should show 0, but it doesn’t. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 25, 2023, 8:02pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/8 "2023-05-25T20:02:43Z")
No, it shouldn’t since you are only resetting the peak which makes the currently used memory the new peak. ---
**Author:** ![ndvbd](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ndvbd/32/56902_2.png) [@ndvbd](https://discuss.pytorch.org/u/ndvbd)\ **Post date:** [May 27, 2023, 4:12pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/9 "2023-05-27T16:12:27Z")
Well, that’s what is happening to me. strange ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 27, 2023, 5:27pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/10 "2023-05-27T17:27:40Z")
In case you misunderstood me: calling `reset_peak_memory` should _not_ always show a 0 memory usage, since memory could still be allocated making it the new peak as seen in my code snippet. ---
**Author:** ![galen\_pereira](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/galen_pereira/32/72015_2.png) [@galen\_pereira](https://discuss.pytorch.org/u/galen_pereira)\ **Post date:** [August 9, 2024, 6:25am UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/11 "2024-08-09T06:25:30Z")
So how to convert that to bytes or MB etc ? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [August 9, 2024, 9:10am UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/12 "2024-08-09T09:10:51Z")
The returned values are given as bytes as described in the [docs](https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html). You can simply convert them to MB by diving them by `1024**2`. ---
**Author:** ![Nancy76](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/b2d939/32.png) [@Nancy76](https://discuss.pytorch.org/u/Nancy76)\ **Post date:** [August 9, 2024, 6:41pm UTC](https://discuss.pytorch.org/t/how-to-calculate-the-gpu-memory-that-a-model-uses/157486/13 "2024-08-09T18:41:46Z")
You need to know the size of each parameter (e.g., 4 bytes for float32) and estimate activation and optimizer states based on your model’s architecture. [Liteblue](https://liteblue.cloud) Tools like TensorBoard or built-in profiling tools in frameworks like PyTorch or TensorFlow can also help.