# Bfloat16 + transformers

**URL:** <https://discuss.pytorch.org/t/bfloat16-transformers/96260>\
**Category:** quantization\
**Created:** [September 14, 2020, 7:03pm UTC](https://discuss.pytorch.org/t/bfloat16-transformers/96260 "2020-09-14T19:03:35Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sam\_Shleifer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_shleifer/32/20402_2.png) [@Sam\_Shleifer](https://discuss.pytorch.org/u/Sam_Shleifer)\
**Post date:** [September 14, 2020, 7:03pm UTC](https://discuss.pytorch.org/t/bfloat16-transformers/96260/1 "2020-09-14T19:03:35Z")

</div>

### Context

In huggingface transformers, the pegasus and t5 models overflow during beam search in half precision.

Models that were originally trained in fairseq work well in half precision, which leads to be believe that models trained in bfloat16 (on TPUS with tensorflow) will often fail to generate with less dynamic range.

I was considering starting a project to further train the models with a penalty for having large activations (to discourage overflow in fp16), but was wondering whether this is duplicative with the pytorch team’s current efforts.

### Specific Questions

- (a) is the snippet below likely to work in a current nightly version?
- (b) are the various kernel implementations “in the works” (and my proposed project won’t be useful in a few months)?
- © Is bfloat16 + cuda a possibility?

### Failing Snippet

The following snippet with tries to run a transformer forward pass in bfloat16:

```auto
from transformers import BartForConditionalGeneration
import torch
model = BartForConditionalGeneration.from_pretrained("sshleifer/distilbart-xsum-12-3")
model = model.to(torch.bfloat16)
input_ids = torch.tensor([[0, 31414, 232, 328, 740, 1140, 12695, 69, 46078, 1588, 2]], dtype=torch.long)
model(input_ids)
# RuntimeError: "LayerNormKernelImpl" not implemented for 'BFloat16'

```

---

<div class="post-metadata">

**Author:** ![jerryzh168](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/32/15217_2.png) [@jerryzh168](https://discuss.pytorch.org/u/jerryzh168)\
**Post date:** [October 9, 2020, 1:28am UTC](https://discuss.pytorch.org/t/bfloat16-transformers/96260/2 "2020-10-09T01:28:04Z")

</div>

Currently quantization supports int8/uint8/int32 and bfloat16 is not covered by pytorch quantization.

@izdeby can you answer the above questions?