Error while trying to build module using pyBind

I am trying to build a module which uses C++ code for non max suppression and using pyBind in order to use it with PyTorch. However, while trying to import the module, I get the following error.

ImportError: /home/chinmay/Desktop/setup/temp/lib/python3.6/site-packages/
special_layers-0.1-py3.6-linux-x86_64.egg/nms_bridge_sample.cpython-36m-x86_64-linux-gnu.so:
undefined symbol:
ZN6caffe28TypeMeta21_typeMetaDataInstanceIN3c107complexIfEEEEPKNS_6
detail12TypeMetaDataEv

I have already created a new virtual environment and installed pyTorch but the error still remains. I tried it on pyTorch 1.5.1 and Ubuntu 18.04