# Are inplace operations faster? **URL:** \ **Category:** Uncategorized\ **Created:** [November 16, 2019, 5:42pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209 "2019-11-16T17:42:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Beinan\_Wang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beinan_wang/32/14160_2.png) [@Beinan\_Wang](https://discuss.pytorch.org/u/Beinan_Wang)\ **Post date:** [November 16, 2019, 5:42pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/1 "2019-11-16T17:42:02Z")
(1) x = a \* x + b (2) x.mul\_(a); x.add\_(b) is (2) faster than (1)? ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [November 16, 2019, 6:20pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/2 "2019-11-16T18:20:12Z")
Mostly not and you don’t get JIT optimisations. Best regards Thomas ---
**Author:** ![Beinan\_Wang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beinan_wang/32/14160_2.png) [@Beinan\_Wang](https://discuss.pytorch.org/u/Beinan_Wang)\ **Post date:** [November 16, 2019, 9:29pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/3 "2019-11-16T21:29:47Z")
So (1) can get JIT optimizations? ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [November 16, 2019, 9:58pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/4 "2019-11-16T21:58:16Z")
On the GPU (1) will get fused into a single kernel (instead of two) by the JIT, saving you 1 of 4 reads and 1 of 2 writes by not writing the intermediate a\*x to memory. A computation like this on sizeable inputs is memory bound, so your saving significantly (if that is a significant part of your computation). Best regards Thomas ---
**Author:** ![Beinan\_Wang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beinan_wang/32/14160_2.png) [@Beinan\_Wang](https://discuss.pytorch.org/u/Beinan_Wang)\ **Post date:** [November 16, 2019, 11:55pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/5 "2019-11-16T23:55:05Z")
Than you very much. Is there any guide on how to write fast PyTorch code like what you said? ---
**Author:** ![tom](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/32/3162_2.png) [@tom](https://discuss.pytorch.org/u/tom)\ **Post date:** [November 17, 2019, 9:56pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/6 "2019-11-17T21:56:00Z")
But so there are bits of advice in various places, and it might depend on what you want to achieve where the best place to look is. NVidia’s [dev blog (to pick a random article)](https://devblogs.nvidia.com/mixed-precision-training-deep-neural-networks/), the [PyTorch article on LSTMs](https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/), I tried to blog a bit about [optimizations](http://lernapparat.de/more-jit-optimizations/) and [a common reduction code pattern](http://lernapparat.de/sinkhorn-kernel/)… In the end, you probably want to benchmark optimizations and gather some experience around what works and does not work. Also, you can always look here at the forums or try to ask with concrete code. We have some cuda questions here every once in a while and I certainly have learnt a lot about GPU programming from people around PyTorch. Best regards Thomas ---
**Author:** ![Beinan\_Wang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/beinan_wang/32/14160_2.png) [@Beinan\_Wang](https://discuss.pytorch.org/u/Beinan_Wang)\ **Post date:** [November 18, 2019, 4:28pm UTC](https://discuss.pytorch.org/t/are-inplace-operations-faster/61209/7 "2019-11-18T16:28:35Z")
Thank you very much!