AssertionError: min nan should be less than max nan

Thank you. After I merged conv and batchnorm. I have solved this problem, but I have encountered this problem in training - aware quantification. Do you have any suggestions?

File “/home/g/anaconda3/lib/python3.7/site-packages/torch/quantization/observer.py”, line 165, in _calculate_qparams
zero_point = qmin - round(min_val / scale)
ValueError: cannot convert float NaN to integer

Is there a problem with my data?

Need to add exception capture?

It was later found that Nan existed in conv2d.weight.
And that’s happened in the process of training.

If your weights got a NaN value, this might be due to a NaN input or a faulty weight update caused by e.g. a high learning rate.
Did you observe the loss during training?
If some weights are exploding, you would usually see a NaN loss.

1 Like

I have the similar problem when model use the training - aware quantification and the training loss is not nan. How to solve it?
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “model-code/resnet34/train_age_quan.py”, line 1130, in forward
x = self.ConvBNReLU1(x)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward
input = module(input)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py”, line 243, in forward
return self.activation_post_process(F.relu(ConvBn2d._forward(self, input)))
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py”, line 95, in _forward
conv = self._conv_forward(input, self.weight_fake_quant(scaled_weight))
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 540, in call
result = self.forward(*input, **kwargs)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/fake_quantize.py”, line 81, in forward
self.scale, self.zero_point = self.calculate_qparams()
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/fake_quantize.py”, line 76, in calculate_qparams
return self.activation_post_process.calculate_qparams()
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/observer.py”, line 481, in calculate_qparams
return self._calculate_per_channel_qparams(self.min_vals, self.max_vals)
File “/mnt/storage1/doris/miniconda3/envs/torch-nightly-py3.6/lib/python3.6/site-packages/torch/quantization/observer.py”, line 150, in _calculate_per_channel_qparams
), “min {} should be less than max {}”.format(min_vals[i], max_vals[i])
AssertionError: min nan should be less than max nan

can you check if values in weights/activations contains nan?