CUDA out of memory problem

1

i tried to use those line

model.eval()
device = torch.device("cuda")
model.to(device)

but got
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 5.80 GiB total capacity; 4.52 GiB already allocated; 4.56 MiB free; 4.53 GiB reserved in total by PyTorch)
in this line
model.to(device)

2

As the error message describes your GPU doesn’t have enough free memory and cannot allocate new memory to push the model to the GPU.
Try to kill other processes which might consume memory and rerun your script.

3

Thanks for replying , i checked the nvidia-smi and didn’t find any process using GPU

4

In that case your script seems to allocate the memory before the model.to() call and you would need to reduce the memory usage.

5

i’m using batch_size =1 how can i clear the memory before i got this problem , please

6

is there any help please

7

You could try to del unneeded tensors before creating the model.
However, it depends on your use case if deleting tensors etc. from the device would actually work or if you would need them.

8

where exactly i should del the tensor , please
the error exactly in this line vec=model(preprocess(image).unsqueeze(0).cuda()) i used a counter after vec1 and got the tensor for the first image then throw this error

9

It seems to error now moved from the model.to() operation to the actual forward pass.
If the only data on the GPU are the model’s parameters and the input tensor, your model might be too large for your device. You could try to use e.g. torch.utils.checkpoint to trade compute for memory.

1 Like
10

thanks a lot for your time but can i extract the features using CPU and used it in my task or it will be a problem due to large size

11

I don’t know how high the peak memory usage for your entire workflow is, but I would assume that your system has more host RAM than the GPU (6GB), so it’s worth a try.

1 Like