- No, that shouldn’t be the case as the compute and accumulate
dtype would still be performed in FP32 even if FP16 inputs are passed.
- If depends on the range of your values and if they are representable. Theoretically it would be possible, e.g. via:
x = torch.tensor(2**(-149)).cuda()
> tensor(1.4013e-45, device='cuda:0')
> tensor(0., device='cuda:0', dtype=torch.float16)
but then I would doubt that the small FP32 values would contribute to your model in any way.
Note that a
GradScaler should be used for mixed-precision training together with
autocast to avoid gradient underflow.
It would be interesting to know more about your use case, i.e. what kind of model “needs” values close to zero to perform properly.