@ ptrblck I apologize in advance . I know I should not mention anyone specific and wait for the comunity to answer but I am in a In a tight spot. It would mean alot if you could help out.
So I have been trying to extract the features of a some pictures for a research case study . when trying to load an image from dataset_loader_ and try to use the model.forward() method for 34 iteration on my dataset_loader_object. this happens even using the cpu instead of gpu:(in this case google colab just crashes) I get a
CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 10.62 GiB already allocated; 832.00 KiB free; 10.66 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
I know this is indicating that I have ran out of cuda memory And here is the model . and the forward decleration :
class vgg16(torch.nn.Module):
def __init__(self, pretrained=True):
super(vgg16, self).__init__()
vgg_pretrained_features = tv.vgg16(pretrained=pretrained).features
self.layerss = torch.nn.Sequential()
for x in range(30):
self.layerss.add_module(str(x), vgg_pretrained_features[x])
self.layerss.eval()
def forward(self, x):
output=[]
for i,layer in enumerate( self.layerss):
# print (i)
x=layer(x)
max_=[img_.max() for chanel in x for img_ in chanel]
min_=[img_.min() for chanel in x for img_ in chanel]
output.append(max_+min_)
return output
model=vgg16()
model.to(device)
But the out of memory erore disapears when I Replace:
max_=[img_.max() for chanel in x for img_ in chanel]
min_=[img_.min() for chanel in x for img_ in chanel]
whit this
x=layer(x)
max_=[img_.max().detach().numpy() for chanel in x for img_ in chanel]
min_=[img_.min().detach().numpy() for chanel in x for img_ in chanel]
but why is this happening ? **Is a tensor obj really that large that 34 containers of type tensor can suck up the space of 10GIB in just matters of seconds ?Am i using the model wrong ?? it’s worth mentioning that the output of the forward function is a list of lists containing tensors
and I used this function to **monitor the cuda memory :
``
def mem_report():
print("CPU RAM Free: " + humanize.naturalsize( psutil.virtual_memory().available ))
GPUs = GPUtil.getGPUs()
for i, gpu in enumerate(GPUs):
print('GPU {:d} ... Mem Free: {:.0f}MB / {:.0f}MB | Utilization {:3.0f}%'.format(i, gpu.memoryFree, gpu.memoryTotal, gpu.memoryUtil*100))
and as mentioned everything looks fine until the for loop : here is the loop code
features=[]
dataset_ =tid2008_dataloader(path,image_name_dir,first_time=True)
start=time.time()
for batch_idx,data in enumerate(dataset_):
data = data.to(device=device)
features.append (model.forward(data))
del data
It’s worth mentioning that the images are the size of 384 * 512*3