Error while post training quantizing the model

I have been getting this error constantly and I need some help on how to solve it.
‘NotImplementedError: Could not run ‘aten::add.out’ with arguments from the ‘QuantizedCPU’ backend’

I’m attaching screenshots of my code + error.


Inference script

 predicted_labels = None
 models_dir = 'xxxxxx'
for model_name in glob.glob(models_dir + '/*.pth'):
     model = CustomResnet(model_name = CFG.model_name, pretrained = False)
     state = torch.load(model_name, 
model =

model = torch.quantization.fuse_modules(model, modules_to_fuse)
model.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(model, inplace = True)
torch.quantization.convert(model, inplace=True)


temp_preds = None

with torch.no_grad():
    for images in tqdm(test_loader, desc=f'Predicting. '):
        images =
        st = time.time()
        predictions = model(images)
        et = time.time()
        predictions = predictions.softmax(1).to('cpu').numpy()
        if temp_preds is None:
            temp_preds = predictions
            temp_preds = np.vstack((temp_preds, predictions))

if predicted_labels is None:
    predicted_labels = temp_preds
    predicted_labels += temp_preds

predicted_labels /= (len(glob.glob(models_dir + ‘/*.pth’)))
test_set[‘predicted’] = predicted_labels.argmax(1)
final_test_accuracy = get_score(test_set[‘target’], test_set[‘predicted’])

print("========================================= PERFORMANCE =============================================")
print(f’Final test accuracy = {final_test_accuracy} ')
print(‘Elapsed time = {:0.4f} milliseconds’.format((et - st) * 1000))

Hi Jino,

What does your model look like? It would appear that you are passing a quantized tensor into a kernel expecting floating point values. Usually the way to fix this is to add a torch.quantization.DeQuantStub before the offending op. This link provides an example of how to fix it.


hey andrew, thanks for writing back, my architecture has been defined the way i posted in the screenshot and dequantstub doesnt work? could you please have a look at the attached?

Hi Jino,

Looking at the output message it seems like the issue is happening maybe because of an addition operation in the pretrained resnet model. That needs to be replaced by FloatFunctional(). The error message has been conveniently truncated so can’t say where is it happening.


Ill give it a try and keep you updated, thanks :slight_smile: