Quantized model inference error related to Mish activation function

quantization
1

I’m tinkering with post-training static quantization in PyTorch by trying out different activation functions on the same model, then I try to quantize it and run inference ( I want to see what are the activations that are supported). For example, I replaced ReLU with leakyReLU on ResNet50 then applied quantization. The inference ran just fine ( it was a bit slower with a 3% accuracy drop but this does not matter as I’m only experimenting). After that, I tried the Mish activation function, the conversion was successful, however, I got the following error during inference:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
 in 
      7 helper.print_size_of_model(resnet)
      8 
----> 9 top1, top5, time_elapsed= helper.evaluate(resnet, criterion, testloader, neval_batches=num_eval_batches)
     10 print('Evaluation accuracy on %d images, top5: %2.2f, top1: %2.2f'%(num_eval_batches * eval_batch_size, top5.avg,top1.avg))
     11 print('time slapsed: %s' % str(datetime.timedelta(seconds=time_elapsed)))

d:\github\PyTorch_CIFAR10\helper.py in evaluate(model, criterion, data_loader, neval_batches, device)
     30         for image, target in data_loader:
     31             image.to(device)
---> 32             output = model(image)
     33             loss = criterion(output, target)
     34             cnt += 1

~\anaconda3\envs\PFE_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

d:\github\PyTorch_CIFAR10\cifar10_models\resnetQ.py in forward(self, x)
    227         x = self.conv1(x)
    228         x = self.bn1(x)
--> 229         x = self.relu(x)
    230         x = self.maxpool(x)
    231         x = self.layer1(x)

~\anaconda3\envs\PFE_env\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

d:\github\PyTorch_CIFAR10\cifar10_models\resnetQ.py in forward(self, x)
     24 
     25     def forward(self, x):
---> 26         x = x * (torch.tanh(torch.nn.functional.softplus(x)))
     27         return x
     28 

RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'QuantizedCPUTensorId' backend. 'aten::empty.memory_format' is only available for these backends: [CPUTensorId, CUDATensorId, MkldnnCPUTensorId, SparseCPUTensorId, SparseCUDATensorId, BackendSelect, VariableTensorId].

the mish layer is defined by:

class Mish(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        x = x * (torch.tanh(torch.nn.functional.softplus(x)))
        return x

Any help in this matter would be greatly appreciated because ultimately, I want to apply quantization on YOLOv4 which relies on Mish as an activation function.

2

Softplus currently does not have a quantized implementation. This error also usually means that a quantized tensor is being passed to a non-quantized function.

For a quick fix, you could add a torch.quantization.DeQuantStub() and torch.quantization.QuantStub() around the areas of the network which cannot be quantized such as Softplus.

The longer term fix would be to add quantization support for Softplus to PyTorch.

3

The quick fix seems reasonable enough. Hopefully it won’t have a significant impact on inference time. I guess I’ll just have to try it out and see. Thank you for your help.

1 Like