# First time run on cuda is slow

**URL:** <https://discuss.pytorch.org/t/first-time-run-on-cuda-is-slow/12482>\
**Category:** Uncategorized\
**Created:** [January 18, 2018, 11:51am UTC](https://discuss.pytorch.org/t/first-time-run-on-cuda-is-slow/12482 "2018-01-18T11:51:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yhs12354123](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/c77e96/32.png) [@yhs12354123](https://discuss.pytorch.org/u/yhs12354123)\
**Post date:** [January 18, 2018, 11:51am UTC](https://discuss.pytorch.org/t/first-time-run-on-cuda-is-slow/12482/1 "2018-01-18T11:51:13Z")

</div>

m = 998  
n = 2473  
a = torch.randn(m\*n).cuda()

st = time.time()  
r = torch.dot(a,a)  
ed = time.time()  
print(ed - st)

this will output 0.09s

st = time.time()  
r = torch.dot(a,a)  
ed = time.time()  
print(ed - st)

This will output 0.0001. This is 100x times faster. What’s the reason? I am on Ubuntu 16.04, pytorch 0.3, cuda 9.

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [January 18, 2018, 12:37pm UTC](https://discuss.pytorch.org/t/first-time-run-on-cuda-is-slow/12482/2 "2018-01-18T12:37:51Z")

</div>

Hi,

Since cuda is initialized lazily in pytorch, the first time you use it, it has a higher runtime.  
Also when doing timings in cuda, you need to manually synchronize because the cuda api is asynchronous.

```python
import torch
import time

m = 998
n = 2473
a = torch.randn(m*n).cuda()

torch.cuda.synchronize()
st = time.time()
r = torch.dot(a,a)
torch.cuda.synchronize()
ed = time.time()
print("initial run")
print(ed - st)

torch.cuda.synchronize()
st = time.time()
r = torch.dot(a,a)
torch.cuda.synchronize()
ed = time.time()
print("normal run")
print(ed - st)

torch.cuda.synchronize()
st = time.time()
r = torch.dot(a,a)
torch.cuda.synchronize()
ed = time.time()
print("normal run")
print(ed - st)

```