Yeah. When I terminate the script it frees the memory but the same thing again when I run it. @smth any thought on that? I want to free that redundant memory when I am running the script so @PTA remounting the gpu during the run does not make sense at all.
I think after torch.cuda.empty_cache() there is a reduced number of remaining memory does make sense. From my experience that should not affect your model training, unless there is a memory leak.
That memory is occupied by something. I want to remove that memory as i want to use that memory. May be trying to put some new tensor in the gpu but that space is still occupied.
@KnHuq we do occupy some base memory of ~400MB per GPU for cuda context, CUDA RNG context, streams etc. and ~200MB per GPU for CuDNN handles etc.
It depends on GPU model as well. The numbers I quoted are for Volta V100 GPU.
I found it would save the very first x even if it has been replaced by conv(x). And it would consume 3 times memory as much as x. Could I free the useless very first x like that?