Hi,
I want to process (with method in torch.nn.functional) some data of torch.half type on CPU in fetching phase of data loader. I know one possible way is to convert the data to torch.float first and then process it. Since I want to feed data of torch.half to my model, I have to convert the data back to torch.half again after processing.
Is there a neater way to do it?
Thanks in advance!
I assume you are running into some unsupported methods using float16 on the CPU during your preprocessing phase and thus would need to transform the data to float32?
If so, then note that your actual model might also run into the same issues as the default mixed-precision support on the CPU is for bfloat16, not float16.
Yes, this is exactly what I ran into. However, my model is supposed to run on the GPU and I should have no concern about these methods right?
What I wanna do is to preprocess the float16 data and keep the data type before moving to GPU for training my model with mixed-precision mode. Maybe there is a better way than float16(original data) → float32(for preprocessing on CPU) → float16(after preprocessing, converted back for GPU training)?