# Pytorch is (much) slower than Tensorflow

**URL:** <https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519>\
**Category:** Uncategorized\
**Created:** [March 10, 2023, 3:40pm UTC](https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519 "2023-03-10T15:40:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eli\_Osherovich](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eli_osherovich/32/58200_2.png) [@Eli\_Osherovich](https://discuss.pytorch.org/u/Eli_Osherovich)\
**Post date:** [March 10, 2023, 3:40pm UTC](https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519/1 "2023-03-10T15:40:00Z")

</div>

I am new to Pytorch and while trying it I felt that familiar models run slower than in Tensorflow.

To illustrate the behavior, I built a very simple fully connected model (single layer) which is available here:

> **[Google Colaboratory](https://colab.research.google.com/drive/1NaSC1X7RtxNKCV4jQEJ_HwNGgVq8viPc?usp=sharing)**

Tensorflow runs three times faster than Pytorch.

Is such a ratio expected? Do I miss something in PT?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 10, 2023, 9:25pm UTC](https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519/2 "2023-03-10T21:25:00Z")

</div>

Your workload is tiny as the model contains a single small linear layer only and you might thus see the overheads of the kernel launches, the data loading etc.  
You could try to use `torch.compile` with `mode='reduce-overhead'`, which would internally use CUDA Graphs, but I doubt this would help since you are using a single layer in the end.

---

<div class="post-metadata">

**Author:** ![Eli\_Osherovich](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eli_osherovich/32/58200_2.png) [@Eli\_Osherovich](https://discuss.pytorch.org/u/Eli_Osherovich)\
**Post date:** [March 12, 2023, 7:10pm UTC](https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519/3 "2023-03-12T19:10:26Z")

</div>

Where/when does this overhead happen? If it happens every epoch (why?) it can somehow explain it. However, it it happens only once, I hardly see how the impact can be so severe over 20 epochs…

I tried compilation with different flags, but it only made things (a bit) worse…

P. S.  
Is there any chance that the developers look into this and, probably, find a way to optimize PT for small problems as well?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 12, 2023, 7:24pm UTC](https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519/4 "2023-03-12T19:24:58Z")

</div>

> [@Eli\_Osherovich](#):
>
> Where/when does this overhead happen? If it happens every epoch (why?) it can somehow explain it. However, it it happens only once, I hardly see how the impact can be so severe over 20 epochs…

> [@ptrblck](#):
>
> and you might thus see the overheads of the kernel launches, the data loading etc.

The data loading is performed by the `DataLoader` and yields a batch in each iteration. If the data loading time exceeds the model training time (which could easily be the case for a single tiny `matmul`) you will see this overhead in each iteration.  
The kernel launch overheads are visible in all kernel launches which are triggered after PyTorch went through its dispatching logic. This overhead is visible the more tiny kernels you want to schedule and CUDA Graphs can help here.

> [@Eli\_Osherovich](#):
>
> Is there any chance that the developers look into this and, probably, find a way to optimize PT for small problems as well?

CUDA Graphs was added for it: [Accelerating PyTorch with CUDA Graphs | PyTorch](https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/)

---

<div class="post-metadata">

**Author:** ![Eli\_Osherovich](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eli_osherovich/32/58200_2.png) [@Eli\_Osherovich](https://discuss.pytorch.org/u/Eli_Osherovich)\
**Post date:** [March 12, 2023, 7:29pm UTC](https://discuss.pytorch.org/t/pytorch-is-much-slower-than-tensorflow/174519/5 "2023-03-12T19:29:43Z")

</div>

Data loading should be identical in both cases – this is an in-memory numpy array.  
Not sure if cuda graphs will be beneficial on CPU… Will they?