# AMP autocast not faster than FP32 **URL:** \ **Category:** mixed-precision\ **Created:** [February 13, 2021, 12:07am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757 "2021-02-13T00:07:39Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![krishansubudhi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishansubudhi/32/20760_2.png) [@krishansubudhi](https://discuss.pytorch.org/u/krishansubudhi)\ **Post date:** [February 13, 2021, 12:07am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/1 "2021-02-13T00:07:39Z")
![image](https://discuss.pytorch.org/uploads/default/original/3X/e/5/e555d4b7436c74e34e7ab53094473034b5aa436c.png) ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 13, 2021, 2:19am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/2 "2021-02-13T02:19:58Z")
CUDA operations are executed asynchronously, so you would have to synchronize the code before starting and stopping the timer. The `torch.utils.benchmark` utility provides this functionality as well as e.g. warmup iterations. ---
**Author:** ![krishansubudhi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishansubudhi/32/20760_2.png) [@krishansubudhi](https://discuss.pytorch.org/u/krishansubudhi)\ **Post date:** [February 14, 2021, 12:15am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/3 "2021-02-14T00:15:49Z")
> [@ptrblck](#): > > torch.utils.benchmark Still not seeing speed up. ```python import torch from torch.cuda.amp import autocast print(torch. __version__ ) !nvidia-smi from transformers import AutoModel model = AutoModel.from_pretrained('bert-base-uncased').to(1) model.train() x=torch.randint(0,1000, (8,128)).to(1) def autocast_forward(model, x, fp16 = False): with autocast(enabled = fp16): return model.forward(x) ``` ```python import torch.utils.benchmark as benchmark t0 = benchmark.Timer( stmt='autocast_forward(model, x, False)', setup='from __main__ import autocast_forward', globals={'x': x, 'model':model}) t1 = benchmark.Timer( stmt='autocast_forward(model, x, True)', setup='from __main__ import autocast_forward', globals={'x': x, 'model':model}) print(t0.timeit(100)) print(t1.timeit(100)) ``` ``` autocast_forward(model, x, False) 19.95 ms 1 measurement, 100 runs , 1 thread autocast_forward(model, x, True) 23.53 ms 1 measurement, 100 runs , 1 thread ``` ```python ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 14, 2021, 5:24am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/4 "2021-02-14T05:24:14Z")
The 15% speedup seems to be low, as the FP16 kernels are not fully saturating the GPU, which is visible in the Nsight Systems profile in the whitespaces between kernel calls. While the FP16 cublas kernels are faster, the overall speedup is lower due to this. ---
**Author:** ![krishansubudhi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishansubudhi/32/20760_2.png) [@krishansubudhi](https://discuss.pytorch.org/u/krishansubudhi)\ **Post date:** [February 14, 2021, 11:23pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/5 "2021-02-14T23:23:00Z")
I don’t see 15% speedup. autocast = false is faster than autocast=true ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [February 14, 2021, 11:25pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/6 "2021-02-14T23:25:23Z")
Ah, thanks. I’ve misunderstood the timings, but saw a minor speedup on an A100 using your benchmark. In any case, it seems the workload is heavily CPU bound. Also there are some unexpected `device_to_device` kernels, so I’ll look into it a bit more. ---
**Author:** ![aminsaied](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aminsaied/32/34786_2.png) [@aminsaied](https://discuss.pytorch.org/u/aminsaied)\ **Post date:** [February 17, 2021, 9:56pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/7 "2021-02-17T21:56:52Z")
For what it’s worth - I reproed this on Tesla M60 GPU and saw the same behavior as OP - that with autocast enabled forward pass is marginally _slower_ than without. ```auto # fp16=True 92.23 ms 1 measurement, 100 runs , 1 thread # fp16=False 79.00 ms 1 measurement, 100 runs , 1 thread ``` I also tried including backward pass, and observed the times were very similar: ```auto model = BertForSequenceClassification.from_pretrained("bert-base-uncased").to(0) tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") inputs = tokenizer("Hello, my dog is cute", return_tensors="pt").to(0) labels = torch.tensor([1]).unsqueeze(0).to(0) def autocast_backward(model, inputs, labels, fp16=False): with autocast(enabled=fp16): outputs = model(**inputs, labels=labels) loss = outputs.loss loss.backward() t0 = benchmark.Timer( stmt="autocast_backward(model, x, False)", setup="from __main__ import autocast_backward", globals={"inputs": inputs, "model": model, "labels": labels,}, ) t1 = benchmark.Timer( stmt="autocast_backward(model, x, True)", setup="from __main__ import autocast_backward", globals={"inputs": inputs, "model": model, "labels": labels,}, ) print(t1.timeit(100)) print(t0.timeit(100)) ``` gave ```auto # fp16=True 304.61 ms 1 measurement, 100 runs , 1 thread # fp16=False 304.99 ms 1 measurement, 100 runs , 1 thread ``` ---
**Author:** ![krishansubudhi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishansubudhi/32/20760_2.png) [@krishansubudhi](https://discuss.pytorch.org/u/krishansubudhi)\ **Post date:** [February 22, 2021, 5:19pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/8 "2021-02-22T17:19:42Z")
@ptrblck any updates on the issue? ---
**Author:** ![jsleep](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jsleep/32/37752_2.png) [@jsleep](https://discuss.pytorch.org/u/jsleep)\ **Post date:** [April 30, 2021, 12:29am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/9 "2021-04-30T00:29:46Z")
Hey @ptrblck , have you gotten a chance to look into this? Anyone working with transformers is better off using apex amp over native amp until there’s some kind of fix for this here. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [April 30, 2021, 5:40am UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/10 "2021-04-30T05:40:48Z")
Sorry for the late reply. I’ve reran the transformer model with a source build using the latest CUDA version and see a speedup on the V100s you are using as: AMP: 16.79ms, FP32: 20.02ms, so I assume cublas might have updated their heuristics and pick faster (TC) kernels. @aminsaied I wouldn’t expect to see a large difference between AMP and FP32 on Maxwell GPUs. @jsleep I don’t agree, as apex/amp is deprecated and used the same autocast list in the then recommended `O1` mode. ---
**Author:** ![jadeantonis](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/977dab/32.png) [@jadeantonis](https://discuss.pytorch.org/u/jadeantonis)\ **Post date:** [August 27, 2021, 7:57pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/11 "2021-08-27T19:57:31Z")
I’m having the same issue on the A6000: ```auto import torch from torch.cuda.amp import autocast from transformers import AutoModel model = AutoModel.from_pretrained('bert-base-uncased').to(1) model.train() x=torch.randint(0,1000, (8,128)).to(1) def autocast_forward(model, x, fast_dtype=None): if fast_dtype is None: return model.forward(x) with autocast(fast_dtype=fast_dtype): return model.forward(x) import torch.utils.benchmark as benchmark t0 = benchmark.Timer( stmt='autocast_forward(model, x)', setup='from __main__ import autocast_forward', globals={'x': x, 'model':model}) t1 = benchmark.Timer( stmt='autocast_forward(model, x, fast_dtype=torch.float16)', setup='from __main__ import autocast_forward', globals={'x': x, 'model':model}) t2 = benchmark.Timer( stmt='autocast_forward(model, x, fast_dtype=torch.bfloat16)', setup='from __main__ import autocast_forward', globals={'x': x, 'model':model}) print(t0.timeit(1000)) # prints 8.91 ms print(t1.timeit(1000)) # prints 9.18 ms print(t2.timeit(1000)) # prints 9.13 ms ``` My goal is to use bf16, not fp16, but the issue is there for both @ptrblck is there any new insight on this? ---
**Author:** ![jadeantonis](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/977dab/32.png) [@jadeantonis](https://discuss.pytorch.org/u/jadeantonis)\ **Post date:** [August 27, 2021, 8:03pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/12 "2021-08-27T20:03:50Z")
and for backward pass: ```auto from torch.utils import benchmark import torch from torch.cuda.amp import autocast from transformers import BertForSequenceClassification, BertTokenizer model = BertForSequenceClassification.from_pretrained("bert-base-uncased").to(0) tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") inputs = tokenizer("Hello, my dog is cute", return_tensors="pt").to(0) labels = torch.tensor([1]).unsqueeze(0).to(0) def autocast_backward(model, inputs, labels, fast_dtype=None): if fast_dtype is None: outputs = model(**inputs, labels=labels) loss = outputs.loss loss.backward() return with autocast(fast_dtype=fast_dtype): outputs = model(**inputs, labels=labels) loss = outputs.loss loss.backward() return t0 = benchmark.Timer( stmt="autocast_backward(model, inputs, labels)", setup="from __main__ import autocast_backward", globals={"inputs": inputs, "model": model, "labels": labels,}, ) t1 = benchmark.Timer( stmt="autocast_backward(model, inputs, labels, torch.float16)", setup="from __main__ import autocast_backward", globals={"inputs": inputs, "model": model, "labels": labels,}, ) t2 = benchmark.Timer( stmt="autocast_backward(model, inputs, labels, torch.bfloat16)", setup="from __main__ import autocast_backward", globals={"inputs": inputs, "model": model, "labels": labels,}, ) print(t0.timeit(100)) # prints 20.52 print(t1.timeit(100)) # prints 22.33 print(t2.timeit(100)) # prints 26.43 :( ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [August 27, 2021, 10:15pm UTC](https://discuss.pytorch.org/t/amp-autocast-not-faster-than-fp32/111757/13 "2021-08-27T22:15:47Z")
Your A6000 would use TF32 by default and would thus already speedup your wokload using TensorCores. [This post](https://github.com/pytorch/pytorch/issues/57806) has additional information, but skip the “channels-last” part since you are working on a language model. Additionally, [this post](https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525/2) discusses a similar issue and provides a profile of the workload.