Pytorch reserves only 1GB on the GPU

Hello all,
I get the Cuda OOM error when I try to run the forward pass on the GPU, with the following as the error message -

RuntimeError: CUDA out of memory. Tried to allocate 5.96 GiB (GPU 0; 7.80 GiB total capacity; 1.18 GiB already allocated; 5.68 GiB free; 1.21 GiB reserved in total by PyTorch)

Firstly, why is pytorch reserving so less memory when my graphics card is of 8GB and second is instead of pytorch reserving memory, is there any way we can tell pytorch to use all the space available.

The model I a trying to run is essentially a retinanet with resnet50 backbone.

I built pytorch from source from the master branch and my torch.__version__ is '1.8.0a0+46b8321'

When the error is raised, 1.18GB are already allocated from the previously executed code and PyTorch tries to allocate 5.96GB, which doesn’t fit into the available 5.68GB.
The rest of the device would most likely be used by the CUDA context (to store the kernels etc.).