Cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518238409320/work/torch/lib/THC/generic/THCStorage.cu:58

Hi,

I make a project where I detect the ground plane from stereo images and for disparity map I want to use many algorithms. One of the algorithms which I want to use is this: (https://github.com/JiaRenChang/PSMNet) This algorithm was developed on NVIDIA Titan X graphic card and I have a NVIDIA GTX 1050 TI 4GB. I make the project work but when I want to train the KITTI 2015 dataset a have the next error:
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1518238409320/work/torch/lib/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last): File "finetune.py", line 193, in <module> main()

File "finetune.py", line 159, in main loss = train(imgL_crop,imgR_crop, disp_crop_L)
File "finetune.py", line 100, in train output1, output2, output3 = model(imgL,imgR)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 357, in __call__ result = self.forward(*input, **kwargs)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 71, in forward return self.module(*inputs[0], **kwargs[0])
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 357, in __call__ result = self.forward(*input, **kwargs)
File "/home/cosmin/licenta/PSMNet/models/stackhourglass.py", line 122, in forward cost0 = self.dres1(cost0) + cost0
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 357, in __call__ result = self.forward(*input, **kwargs)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward input = module(input)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 357, in __call__ result = self.forward(*input, **kwargs)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward input = module(input)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 357, in __call__ result = self.forward(*input, **kwargs)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/modules/batchnorm.py", line 37, in forward self.training, self.momentum, self.eps)
File "/home/cosmin/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 1013, in batch_norm return f(input, weight, bias)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518238409320/work/torch/lib/THC/generic/THCStorage.cu:58
The python file that I run for training is finetune.py
My question is that code can run on my graphic card or don’t have enough power and memory?
And if can run what can I do to to remove the error with cuda runtime error(2)?

What’s your nvidia-smi output? Does it look like you’re running out of memory?

This is the output of my nvidia-smi:
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1192 G /usr/lib/xorg/Xorg 187MiB |
| 0 2059 G compiz 63MiB |
| 0 2271 G …-token=6A135D8F234A0DBE5B71ECB1693F9899 48MiB |
| 0 3002 C python 3717MiB |
±----------------------------------------------------------------------------+

Yes, look like running out of memory. I have 4 GB of memory on my graphic card.

You can try to decrease the batch size.