hi!
I was attempting to train some part of my model with fp8 and fp16, and i’d like to ask:
is there a way to achieve the 1-5-2 fp8 in some calculation, such as plus or mul?
normally the fp16 is 1-5-10, is it possible to make it 1-6-9 or 1-4-11?
thanks!
I don’t think there is an easy way of changing the exponent and mantissa of the underlying values.
You could of course try to simulate these number formats, but I’m not sure how much work that would be.
yeah I found it difficult to change the exp and mantissa. now I tried to simulate the errors.
Thanks anyway
hi,and I have one more question
is there a easy way to extract the [neg, exponent, mantissa] of a torch.tensor in cuda? just like in numpy can do view() and some bitwise operation, which cannot work on GPU device.
If you have the float data in your CUDA method, you could use e.g. frexpf or some bit operations on the bits. There should be some examples, how this would work in C++/C on StackOverflow.