Generate equal random variable even if dtype is different

Seeing that we get the same bernoulli_ after uniform_, I think the seed is managed regardless of whether we are operating on float32 or float16.
By the way, unlike the result I got from the code, I want to make the normal_() obtained from the tensor of float16 the same as the value obtained from the normal_() generated from float32 (without type casting).

Any ideas?

import torch
torch.manual_seed(42)
print(torch.empty(10, dtype=torch.float32).uniform_())
print(torch.empty(10, dtype=torch.bool).bernoulli_())
torch.manual_seed(42)
print(torch.empty(10, dtype=torch.float16).uniform_())
print(torch.empty(10, dtype=torch.bool).bernoulli_())
tensor([0.8823, 0.9150, 0.3829, 0.9593, 0.3904, 0.6009, 0.2566, 0.7936, 0.9408,
        0.1332])
tensor([ True, False, False, False, False,  True, False, False, False, False])
tensor([0.5498, 0.7124, 0.4199, 0.6318, 0.5518, 0.5347, 0.8418, 0.5098, 0.7998,
        0.0591], dtype=torch.float16)
tensor([ True, False, False, False, False,  True, False, False, False, False])

I don’t know how this should work, since you cannot represent all float32 values in float16. Sampling normal_ values in float16 thus has to stick to valid float16 values.

Assuming R16/R32 is a tensor with normal_ operation performed on a tensor of float16/float32 respectively, I expected the following to be correct.

diff = (R16 + (R16.float() -  R32).half())-R16
assert diff.max().item() == 0

I have not tried it on other gpu, but I have confirmed that the condition is satisfied in the code executed on MX250@2GB. However, the CPU behaves differently.

import torch
torch.manual_seed(42)
R32 = torch.empty(10, dtype=torch.float32, device="cuda").uniform_()
torch.manual_seed(42)
R16 = torch.empty(10, dtype=torch.float16, device="cuda").uniform_()

diff_fp16 = (R16 + (R16.float() -  R32).half())-R16
assert diff_fp16.max().item() == 0, diff_fp16.max().item()

In your current code snippet it seems you are expecting that:

fl32_eps  R16.float() -  R32
fp16_eps  = fp32_eps.half()
fp16_diff = R16 - fp16_eps

underflows and returns R16 again since diff is expected to be a perfect 0..
I don’t understand how this is related to:

I want to make the normal_() obtained from the tensor of float16 the same as the value obtained from the normal_() generated from float32

I want the following two tensors to have the same value.

  1. Initialize the tensor of float32 to normal_ (R32) and then typecast to float16 (R32to16)
  2. Initialize the tensor of float16 to normal_ (R16)

Reflecting the point that it is impossible to say that they have the same value here, I wanted to make sure that both tensors have the same value when expressed as a float16. (R16 == R32to16)

So, the added snippet is that float16 cannot handle the full range of float32, so I compare R16 and R32to16 and check whether R16 is the most accurate float16 expressing R32 by applying eps.