# Torch.cuda.amp cannot speed up on A100

**URL:** <https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525>\
**Category:** mixed-precision\
**Created:** [May 7, 2021, 6:55am UTC](https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525 "2021-05-07T06:55:34Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 7, 2021, 8:00am UTC](https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525/2 "2021-05-07T08:00:39Z")

</div>

I’m not familiar with this model, but note that you are already using TensorCores on the A100, since TF32 is enabled by default. With a proper synchronization I get a runtime of:

```auto
FP32: 102s
TF32: 50s
AMP: 47s

```

Based on the profile it also seems that the majority or kernels are (vectorized/unrolled) elementwise kernels and sporadically a cublas kernel is called:  
FP32:

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/2/3/23a6592eb781ef6c91fc57cbd4a908d5f38b41c9.png)  
FP16:  
 ![image](https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0f469fe204357a57a50e6f51d84f91fabac07ed.png)

Based on the [Ops that can autocast to float16](https://pytorch.org/docs/stable/amp.html#ops-that-can-autocast-to-float16) it seems that no many of these operations are used.

---

_[View the full topic](https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525)._