# Error in \`python': free(): invalid pointer: When using model.cuda() on AWS Instance

**URL:** <https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857>\
**Category:** vision\
**Created:** [April 14, 2017, 5:51am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857 "2017-04-14T05:51:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sujithv28](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sujithv28/32/452_2.png) [@sujithv28](https://discuss.pytorch.org/u/sujithv28)\
**Post date:** [April 14, 2017, 5:51am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/1 "2017-04-14T05:51:02Z")

</div>

For some reason when I set model.cuda() along with the other examples I get the following error:  
\*\*\* Error in `python’: free(): invalid pointer: 0x00007f8af6c2bae0 \*\*\*

However when I no longer set model.cuda() I get no pointer free errors and the model trains fine. Do I have to set .cuda() on every single variable including the criterion?

I am using python2 and using the Udacity Tensorflow g2.2xlarge instance on Amazon AWS.

Here is a link to my code:

> <https://github.com/sujithv28/Deep-LeafSnap/blob/master/model.py>

Thank you!

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [April 14, 2017, 8:11am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/2 "2017-04-14T08:11:08Z")

</div>

Hi,

The `.cuda()` operation is not inplace for tensors, your should do `input = input.cuda()`.  
That being said it should just raise an error, not fail like that.

---

<div class="post-metadata">

**Author:** ![sujithv28](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sujithv28/32/452_2.png) [@sujithv28](https://discuss.pytorch.org/u/sujithv28)\
**Post date:** [April 14, 2017, 8:53am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/3 "2017-04-14T08:53:33Z")

</div>

I changed everything to `.cuda()` now but this is the error I get instead:

```auto
THCudaCheck FAIL file=/py/conda-bld/pytorch_1490983232023/work/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
Traceback (most recent call last):
  File "model.py", line 244, in <module>
    train(train_loader, model, criterion, optimizer, epoch)
  File "model.py", line 118, in train
    loss.backward()
  File "/home/ubuntu/anaconda3/lib/python3.5/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
RuntimeError: cuda runtime error (2) : out of memory at /py/conda-bld/pytorch_1490983232023/work/torch/lib/THC/generic/THCStorage.cu:66

```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [April 14, 2017, 9:02am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/4 "2017-04-14T09:02:39Z")

</div>

You don’t have enough memory on the GPU, you may want to reduce the batch size.

---

<div class="post-metadata">

**Author:** ![iammohitchhabra](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/848f3c/32.png) [@iammohitchhabra](https://discuss.pytorch.org/u/iammohitchhabra)\
**Post date:** [May 23, 2017, 4:33pm UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/5 "2017-05-23T16:33:59Z")

</div>

I get the same error on running the cartpole example with cuda. Hower as mentioned above without cuda it runs fine. The error persists even on reducing the batch\_size to 2. Any solutions?

---

<div class="post-metadata">

**Author:** ![iammohitchhabra](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/848f3c/32.png) [@iammohitchhabra](https://discuss.pytorch.org/u/iammohitchhabra)\
**Post date:** [May 23, 2017, 5:30pm UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/6 "2017-05-23T17:30:18Z")

</div>

sudo apt-get install libtcmalloc-minimal4  
export LD\_PRELOAD="/usr/lib/libtcmalloc\_minimal.so.4"

Fixes the error.

---

<div class="post-metadata">

**Author:** ![mbednarski](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbednarski/32/29094_2.png) [@mbednarski](https://discuss.pytorch.org/u/mbednarski)\
**Post date:** [August 3, 2017, 5:48pm UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/7 "2017-08-03T17:48:28Z")

</div>

It works, thanks. But do you know why?

---

<div class="post-metadata">

**Author:** ![jupiter](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/53a042/32.png) [@jupiter](https://discuss.pytorch.org/u/jupiter)\
**Post date:** [August 10, 2017, 8:32pm UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/8 "2017-08-10T20:32:48Z")

</div>

Indeed it solves the “invalid pointer error”! Can anyone explain why?

---

<div class="post-metadata">

**Author:** ![Jared\_77](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jared_77/32/5293_2.png) [@Jared\_77](https://discuss.pytorch.org/u/Jared_77)\
**Post date:** [July 4, 2018, 12:16am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/9 "2018-07-04T00:16:21Z")

</div>

Is there any solution to this for someone on an academic institution cluster without sudo privileges?

---

<div class="post-metadata">

**Author:** ![MJ-Osis](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mj-osis/32/48557_2.png) [@MJ-Osis](https://discuss.pytorch.org/u/MJ-Osis)\
**Post date:** [August 16, 2022, 7:28am UTC](https://discuss.pytorch.org/t/error-in-python-free-invalid-pointer-when-using-model-cuda-on-aws-instance/1857/10 "2022-08-16T07:28:13Z")

</div>

This solved the problem, but after a few more epochs it crashed again. Any more suggestions?
