Out of memory error when using cuda_profiler

1

I’m trying to bottleneck my code.
The code is able to run under cProfiler and autograd cpu profiler. However it will raise out of memory error when running under cuda profiler.

And there’s no error stack; I got this:
terminate called after throwing an instance of ‘std::runtime_error’
what(): /opt/pytorch/torch/csrc/autograd/profiler.h:53: out of memory

BTW: I also monitored the gpu memory usage: the memory usage was shown to be 800mb while the whole ram size is 11g.

1 Like
2

Did you find any solution? For me, the cuda profiler just eats all the RAM that I have (32GB), it never actually fully run out of memory, but it fills it almost completely and I don’t get any results back. I waited for a couple of hours, but that shouldn’t take that long.

3

I didn’t know why, but I avoid this error by replacing some torch operations to numpy operations. (mostly involving indexing.)