I noticed that torch.float16’ s smallest positive number 2**(-24). However, IEEE 754 set the smallest non-zero positive number as 2**(-14). (when not considering denormal representation)
How does torch.float16 work? Is it different from IEEE754 ? Do they apply different exponent bias (a.k.a. zero offset) according to the tensor’s distribution?
The below code shows what I have tried:
x = torch.tensor((2**(-24)*1.))
x
tensor(5.9605e-08)
x.half()
tensor(5.9605e-08, dtype=torch.float16)
Hi Mary!
You have it right.
No. torch.float16 is standard IEEE 754. (It actually depends on your
hardware, but for all practical purposes, all* hardware is IEEE 754.)
Your only problem is that the your example of 2**-24 gives your a
subnormal number (as per IEEE 754).
Consider:
>>> import torch
>>> print (torch.__version__)
1.13.1
>>> import numpy as np
>>>
>>> torch.finfo (torch.half).smallest_normal
6.103515625e-05
>>> 2**-14
6.103515625e-05
>>>
>>> torch.tensor (5.9605e-08).half()
tensor(5.9605e-08, dtype=torch.float16)
>>> 2**-24
5.960464477539063e-08
>>>
>>> torch.tensor (torch.finfo (torch.half).smallest_normal, dtype = torch.half)
tensor(6.1035e-05, dtype=torch.float16)
>>> torch.tensor (torch.finfo (torch.half).smallest_normal, dtype = torch.half) / 2
tensor(3.0518e-05, dtype=torch.float16)
>>>
>>> torch.tensor (5.9605e-08).half()
tensor(5.9605e-08, dtype=torch.float16)
>>> torch.tensor (5.9605e-08).half() / 2
tensor(0., dtype=torch.float16)
>>>
>>> np.finfo (np.half).smallest_normal
6.104e-05
>>> np.finfo (np.half).smallest_subnormal
6e-08
*) Well, there is always NVIDIA’s dreaded TensorFloat-32 format …
Best.
K. Frank