Got error 'Cannot allocate memory'

1

Code like this:

trainset = COCODataset('data.txt', (416,416), True, is_debug=True)
	trainloader = DataLoader(trainset, batch_size=8,shuffle=False, num_workers=4)

	net = Mynet(file)
	if gpu:
		net.cuda()
	optimizer = optim.SGD(net.parameters(), lr=0.01)
	for epoch in range(Epochs):
		for i, sample in enumerate(trainloader):
			images, labels = sample['image'], sample['label']

I got an error here for i, sample in enumerate(trainloader): :
OSError: [Errno 12] Cannot allocate memory
What’s that mean?My GPU memory-usage is 1283MiB / 8118MiB

2

The error message seems to point to your RAM, not the GPU memory.
Could you check it with free -h and see how much memory is left?

This particular error message can be triggered, if new processes would like to spawn / fork and there is not enough memory left.

3

how can I trigger such particular error message?
Is there any code implementation in pytorch for that?
I obtained this memmory allocation error in loading images from KITTI dataset.

4

If you are trying to load all images into your RAM, you might trigger this error.
Did you observe the RAM usage?

5

$ free -h shows the available memory decreasing to almost ZERO and this is the error eventually I get:

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 12976128000 bytes. Error code 12 (Cannot allocate memory

what’s a better approach to load images or read images without allocating RAM ?

6

Usually you would lazily load the samples using a Dataset and wrap it in a DataLoader to create batches, shuffle the data, use multiple workers to load each batch etc.

Have a look at the Data loading tutorial for more information on how to write a custom Dataset.

7

try to set overcommit_memory to 0 in /etc/sysctl.conf

8

i also meet the memory leak problem. but i dont know how to correct it…

Traceback (most recent call last):
  File "main_multi_model_test.py", line 169, in <module>
    main()
  File "main_multi_model_test.py", line 143, in main
    val_loss, val_acc, val_bacc = validate(model, val_loader, criterion, taskNum, weights=attrWeights)
  File "../validate.py", line 30, in validate
    for i, (inputs, labels) in enumerate(val_loader):
  File "/home/user1/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 193, in __iter__
    return _DataLoaderIter(self)
  File "/home/user1/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 469, in __init__
    w.start()
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/user1/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory