# Is torch.float16 different from IEEE754? torch.float16 shows smaller value than 2**(-15)

I noticed that torch.float16’ s smallest positive number 2**(-24). However, IEEE 754 set the smallest non-zero positive number as 2**(-14). (when not considering denormal representation)

How does torch.float16 work? Is it different from IEEE754 ? Do they apply different exponent bias (a.k.a. zero offset) according to the tensor’s distribution?

The below code shows what I have tried:

x = torch.tensor((2**(-24)*1.))
x
tensor(5.9605e-08)
x.half()
tensor(5.9605e-08, dtype=torch.float16)

Hi Mary!

You have it right.

No. `torch.float16` is standard IEEE 754. (It actually depends on your
hardware, but for all practical purposes, all* hardware is IEEE 754.)

Your only problem is that the your example of `2**-24` gives your a
subnormal number (as per IEEE 754).

Consider:

``````>>> import torch
>>> print (torch.__version__)
1.13.1
>>> import numpy as np
>>>
>>> torch.finfo (torch.half).smallest_normal
6.103515625e-05
>>> 2**-14
6.103515625e-05
>>>
>>> torch.tensor (5.9605e-08).half()
tensor(5.9605e-08, dtype=torch.float16)
>>> 2**-24
5.960464477539063e-08
>>>
>>> torch.tensor (torch.finfo (torch.half).smallest_normal, dtype = torch.half)
tensor(6.1035e-05, dtype=torch.float16)
>>> torch.tensor (torch.finfo (torch.half).smallest_normal, dtype = torch.half) / 2
tensor(3.0518e-05, dtype=torch.float16)
>>>
>>> torch.tensor (5.9605e-08).half()
tensor(5.9605e-08, dtype=torch.float16)
>>> torch.tensor (5.9605e-08).half() / 2
tensor(0., dtype=torch.float16)
>>>
>>> np.finfo (np.half).smallest_normal
6.104e-05
>>> np.finfo (np.half).smallest_subnormal
6e-08
``````

*) Well, there is always NVIDIA’s dreaded TensorFloat-32 format …

Best.

K. Frank