I’m using https://docs.pytorch.org/memory_viz (thanks for the amazing tool) to debug my OOM issue and finding it confused about the output.
Take the screenshot below as an example, we focus on memory allocation A and B.
As the red box highlighted, there is 22.8 GB memory left after allocation A is allocated.
However, after allocation B is allocated, the total memory became 21.2 GB, which seems to contradict the graph as B should be top of A. The y-axis also suggests that after B is allocated, the total memory used is around 22.7 GB. It leads to another question: how to know the memory peak value? For instance, what’s the maximum memory of the graph below? Is it 27.3 GB(inferred from the y-axis) or 25.4 GB(read from the log)?
It’s worth mentioning that allocation B lives longer than A. I wonder if the total memory used after allocation counts the lowest memory watermark.
Thanks in advance for your insights.
cc @zdevito , @aaronenyeshi