# Bfloat16 has worse performance than float16 for conv2d **URL:** \ **Category:** Uncategorized\ **Created:** [June 17, 2022, 5:46am UTC](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373 "2022-06-17T05:46:53Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![StevenTsai](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dbc845/32.png) [@StevenTsai](https://discuss.pytorch.org/u/StevenTsai)\ **Post date:** [June 17, 2022, 5:46am UTC](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373/1 "2022-06-17T05:46:53Z")
Hi, I just compared the performance of my model with different parameter data types, and I found that using bfloat16 would get worse performance than float16. Is it expected or not? For the further investigation, I used simple models to compare the performance of bfloat16 and float16. exp1. > > > input = torch.randn(20, 16, 500, 1000) > > > m = nn.Conv2d(16, 33, 3, stride=2) > > > for \_ in range(1000): > > > output = m(input) exp2. > > > input = torch.randn(10000, 10000) > > > m = nn.Linear(10000, 20000) > > > for \_ in range(1000): > > > output = m(input) For exp1, the execution time of float16/bfloat16/float32 was 2.1/ **3.8** /3.2 s. while for exp2, the execution time of float16/bfloat16/float32 was 20.1/ **19.5** /33.8 s. For conv2, the performance of bfloat16 was even was than float32. Is this expected? By the way, all my experiments were done by A100 GPUs (80 GB) with torch 1.11.0, cuda 11.1.3, NCCL 2.8.4 ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [June 17, 2022, 7:11am UTC](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373/2 "2022-06-17T07:11:38Z")
> [@StevenTsai](#): > > By the way, all my experiments were done by A100 GPUs (80 GB) with torch 1.11.0, cuda 11.1.3, NCCL 2.8.4 In your code snippet you are not using the GPU, so I’m unsure if the code snippet is wrong or if you are measuring the CPU performance and the GPU information is irrelevant. If you are using the device, then note that synchronizations are needed. To enable bfloat16 support for conv layers, you would have to enable the cuDNN v8 API via `TORCH_CUDNN_V8_API_ENABLED=1` and would need a build with this API enabled via `USE_EXPERIMENTAL_CUDNN_V8_API=1` (the current nightly binaries should already build with it). ---
**Author:** ![StevenTsai](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dbc845/32.png) [@StevenTsai](https://discuss.pytorch.org/u/StevenTsai)\ **Post date:** [June 18, 2022, 1:13am UTC](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373/3 "2022-06-18T01:13:22Z")
Actually, my original code was, exp2. > > > input = torch.randn(10000, 10000).to(device=\_gpu\_device, dtype=\_type) > > > m = nn.Linear(10000, 20000).to(device=\_gpu\_device, dtype=\_type) > > > torch.cuda.synchronize() > > > import time > > > t = time.time() > > > for \_ in range(100): > > > output = m(input) > > > torch.cuda.synchronize() > > > time\_elapse = time.time() - t > > > print(f"time\_elapse = {time\_elapse}") Exp1 is written in the same way. Since I wanted to make my exp easy to be read, I just simplified it. Thanks for you comments, I will tried environment variables you mentioned. ---
**Author:** ![StevenTsai](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dbc845/32.png) [@StevenTsai](https://discuss.pytorch.org/u/StevenTsai)\ **Post date:** [June 23, 2022, 2:25am UTC](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373/4 "2022-06-23T02:25:49Z")
@ptrblck , I have tried to set the environment variables but found no gain for bfloat16 performance. On the other hand, I profiled my experiment using Nvidia Nsight Systems. The diagram showed that different kernels were applied for float16 and bfloat16. Did I run model with bfloat16 correctly? bfloat16 profiling result ![bfloat16](https://discuss.pytorch.org/uploads/default/original/3X/3/5/35d2741924eafe2b2d66021a8008c5302bb74118.png) ---
**Author:** ![StevenTsai](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dbc845/32.png) [@StevenTsai](https://discuss.pytorch.org/u/StevenTsai)\ **Post date:** [June 23, 2022, 2:26am UTC](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373/5 "2022-06-23T02:26:49Z")
@ptrblck float16 profiling result ![float16](https://discuss.pytorch.org/uploads/default/original/3X/8/5/856b42ed1d699a4139f863548a61ad02ec60c0b9.png)