I think it would be better to use if pytorch supports fp4(e3m0, e2m1,…), fp3(e1m1, e2m0,…) some types like that.
forward and backward, several caculation supports while we using… like fp32 or bf16. It will be far better to use when users need PTQ or QAT or making small models.
So I wonder if pytorch has plan to make fp4 and fp3 data type officially in pytorch.
I think 4 bit float and 3 bit float is needed and fp8 does not seem to support ‘any kinds of calculation in any cpu or gpu device’. ('un’limited support)
-torch.float8_e4m3fn and torch.float8_e5m2 implement the spec for 8-bit floating point types from [2209.05433] FP8 Formats for Deep Learning. The op support is very limited.
I think many users want a new update about that…
Of course there will be someone who needs smaller…
And I think they (smaller float types) would be soon used widely…
As I’ve been watching someone who made some small data type code and telling to some company like nvidia…
I want pytorch provides unlimited support on ‘officially optimized’ reasonable small float point data types.
Not so excellent in english.
Thanks and best regards…