# When to set pin\_memory to true? **URL:** \ **Category:** vision\ **Created:** [June 14, 2018, 8:48pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723 "2018-06-14T20:48:31Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kindlychung](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kindlychung/32/3679_2.png) [@kindlychung](https://discuss.pytorch.org/u/kindlychung)\ **Post date:** [June 14, 2018, 8:48pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/1 "2018-06-14T20:48:31Z")
From the imagenet [example](https://github.com/pytorch/examples/blob/master/imagenet/main.py#L133-L145): ```auto train_loader = torch.utils.data.DataLoader( train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None), num_workers=args.workers, pin_memory=True, sampler=train_sampler) val_loader = torch.utils.data.DataLoader( datasets.ImageFolder(valdir, transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), normalize, ])), batch_size=args.batch_size, shuffle=False, num_workers=args.workers, pin_memory=True) ``` Both data loaders set `pin_memory` to true, while the default value is false. Are there good reasons to do this? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [June 14, 2018, 8:52pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/2 "2018-06-14T20:52:31Z")
If you load your samples in the `Dataset` on CPU and would like to push it during training to the GPU, you can speed up the host to device transfer by enabling `pin_memory`. This lets your `DataLoader` allocate the samples in page-locked memory, which speeds-up the transfer. You can find more information on the [NVIDIA blog](https://devblogs.nvidia.com/how-optimize-data-transfers-cuda-cc/). ---
**Author:** ![kindlychung](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kindlychung/32/3679_2.png) [@kindlychung](https://discuss.pytorch.org/u/kindlychung)\ **Post date:** [June 15, 2018, 8:15am UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/3 "2018-06-15T08:15:53Z")
Thanks. This means `pin_memory` should always be true if you are training on a Nvidia GPU, right? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [June 15, 2018, 10:25am UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/4 "2018-06-15T10:25:10Z")
Generally, yes, if you are loading your data on the CPU. It won’t work if your dataset is really small and you push it already to the GPU in the `Dataset`, but that’s pretty obvious. ---
**Author:** ![cakeeatingpolarbear](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/c0e974/32.png) [@cakeeatingpolarbear](https://discuss.pytorch.org/u/cakeeatingpolarbear)\ **Post date:** [December 27, 2018, 6:51am UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/5 "2018-12-27T06:51:18Z")
If I store the paths to my images in my dataset (as I can’t fit my dataset into RAM) and only load the images in the **getitem** function is it still beneficial to use pin memory? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [December 27, 2018, 1:34pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/6 "2018-12-27T13:34:40Z")
Yes. As far as I know, that’s the exact use case for speeding up the transfer between the host and device. ---
**Author:** ![Ravi\_Tej\_Akella](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_tej_akella/32/9912_2.png) [@Ravi\_Tej\_Akella](https://discuss.pytorch.org/u/Ravi_Tej_Akella)\ **Post date:** [September 27, 2019, 3:30pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/7 "2019-09-27T15:30:18Z")
@ptrblck Surprisingly, I am getting better speed-ups with pin\_memory=False. Also, the CPU memory usage has also reduced by 2 or 3 folds with pin\_memory=False. Just curious what might be the reason. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [September 27, 2019, 8:03pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/8 "2019-09-27T20:03:57Z")
We had a recent [issue](https://github.com/pytorch/pytorch/issues/25010), which should have been fixed by now. Are you seeing the same behavior using the latest nightly build? ---
**Author:** ![Ravi\_Tej\_Akella](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_tej_akella/32/9912_2.png) [@Ravi\_Tej\_Akella](https://discuss.pytorch.org/u/Ravi_Tej_Akella)\ **Post date:** [September 28, 2019, 12:43pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/9 "2019-09-28T12:43:32Z")
This issue occurs in Pytorch 1.1.0 version. I have not tried out the latest nightly version. My device has CUDA 9.0 (I cannot upgrade CUDA), which I guess is not supported by the latest nightly version. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [September 28, 2019, 9:30pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/10 "2019-09-28T21:30:00Z")
The binaries ship with their own CUDA and cudnn versions so that you don’t have to install CUDA locally (just the NVIDIA drivers). ---
**Author:** ![Ravi\_Tej\_Akella](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_tej_akella/32/9912_2.png) [@Ravi\_Tej\_Akella](https://discuss.pytorch.org/u/Ravi_Tej_Akella)\ **Post date:** [September 28, 2019, 10:38pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/11 "2019-09-28T22:38:02Z")
Oh okay. Thanks for the help 🙂 ---
**Author:** ![PySimon](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e5b9ba/32.png) [@PySimon](https://discuss.pytorch.org/u/PySimon)\ **Post date:** [October 22, 2019, 6:49pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/12 "2019-10-22T18:49:30Z")
Hi, when setting pin\_memory=True, many more processes are forked than with pin\_memory=False, even with num\_workers=0 in all cases. What is the reason for that? For me pin\_memory=False is much faster in this case. The tensors come out of my dataloader as CPU tensors and before running my network I call to(device=MY\_GPU\_ID). Is that wrong? How else should I specify on which GPU I want the tensor to be? Pytorch is 1.2.0 Best, Simon ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [October 22, 2019, 7:24pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/13 "2019-10-22T19:24:55Z")
We had an issue using too many threads when `pin_memory=True` was set recently, which should be fixed in the latest release. Could you update your PyTorch installation and check the CPU usage again? ---
**Author:** ![Shubhankar](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b9bd4f/32.png) [@Shubhankar](https://discuss.pytorch.org/u/Shubhankar)\ **Post date:** [November 6, 2019, 1:22am UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/14 "2019-11-06T01:22:36Z")
On PyTorch 1.13 I am still getting faster runtimes with `pinned_memory=False` and using num\_workers=20. I have 4 NVIDIA P100 with 256 GB IBM Power8. ---
**Author:** ![AndreaCatania](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ba9def/32.png) [@AndreaCatania](https://discuss.pytorch.org/u/AndreaCatania)\ **Post date:** [December 8, 2019, 1:02pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/15 "2019-12-08T13:02:47Z")
The tensors that comeout from the `DataLoader` are on CPU, (at least this is what `print(t.device)` says), even if: ```auto num_workers=1, pin_memory=True ``` What’s the correct way to upload these directly on the GPU? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [December 8, 2019, 4:38pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/16 "2019-12-08T16:38:51Z")
`pin_memory=True` uses pinned host memory for faster copies as described [here](https://devblogs.nvidia.com/how-optimize-data-transfers-cuda-cc/). Push your tensors via `x =x.to('cuda')` to the device. ---
**Author:** ![devforfu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ecd19e/32.png) [@devforfu](https://discuss.pytorch.org/u/devforfu)\ **Post date:** [August 1, 2020, 7:16pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/17 "2020-08-01T19:16:51Z")
Sorry for bumping up this rather old thread. But how does this work with two and more GPUs? In my use case, I have two experiments that run in parallel, two different models. I use `pin_memory=True` in one of my experiments with `cuda:0` device. Then I run one more experiment but with `cuda:1`, and it fails with OOM error because of making attempt to allocate the memory on another GPU. Is it an expected behavior? ---
**Author:** ![raharth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raharth/32/14974_2.png) [@raharth](https://discuss.pytorch.org/u/raharth)\ **Post date:** [September 2, 2020, 12:17pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/18 "2020-09-02T12:17:38Z")
Hi! I stumbled upon this thread, today wondering about the slowing down of my training.It looks as if the same problem still exists. When setting pin\_memory=True it processes roughly half as many iterations as if set to False. I’m using CUDA10.2 and PyTorch 1.6.0 on Win10 on a K80. My dataset is fairly small with just around 300MB, could that be the reason? ---
**Author:** ![Emre\_Yalcinoglu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emre_yalcinoglu/32/31705_2.png) [@Emre\_Yalcinoglu](https://discuss.pytorch.org/u/Emre_Yalcinoglu)\ **Post date:** [December 21, 2020, 6:10pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/19 "2020-12-21T18:10:29Z")
Thanks for the answers. I wanted to clarify one more thing: I have a setup where I push the model to the GPU, and then push the input tensors to the GPU **only** during the training like so: ```auto def train(model, dataloader): for epoch in range(num_of_epochs): for image_batch, label_batch in dataloader: image_batch, label_batch = image_batch.to(device), label_batch.to(device) optimizer.zero_grad() ... ``` However, I am not observing any speed-up. Not even after the first epoch, which I thought how `pin_memory=True` would affect the process. So I have two further questions: 1. Is pushing to the device during ` __getitem__ ` the best-practice? In SO, I saw a comment stating otherwise. If not, then was my example, pushing during training loop, correct? 2. Do we have to state `num_workers` in order to get performance help from `pin_memory=True`? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [December 23, 2020, 6:49pm UTC](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723/20 "2020-12-23T18:49:20Z")
1. If you want (and/or have to) to lazily load the data, then I would stick with the `Dataset` approach and load each sample in the ` __getitem__ `. You could e.g. preload the complete MNIST dataset and might even be able to push it to the GPU before the training begins (which would give you a performance benefit during training), but that’s of course not always possible especially if you are working with larger datastes. 2. Try to use `non_blocking=True` in the `to()` operation. [Next page](https://discuss.pytorch.org/t/when-to-set-pin-memory-to-true/19723.md?page=2)