Typecasting network and data to HalfTensor results in all parameters = NaN

[tom (Thomas V)]

The (lack of) numeric stability of half can bite you fairly easily. For example, batch norm basically doesn’t seem to work well with half. There is some more discussion in the mail and thread linked below.

Best regards

Thomas

Training with Half Precision

It works, but you want to make sure that the BatchNormalization layers use float32 for accumulation or you will have convergence issues. You can do that by something like: model.half() # convert to half precision for layer in model.modules(): if isinstance(layer, nn.BatchNorm2d): layer.float() Then make sure your input is in half precision. Christian Sarofeen from NVIDIA ported the ImageNet training example to use FP16 here: We’d like to clean-up the FP16 support to make it more a…