Building wheels with pytorch extensions

I am trying to build a manylinux wheel containing pytorch extensions using CUDA. Everything works fine except for patchelf’ing libcudart in the so files. After running auditwheel --repair on the wheel, the compiled extensions have been patched as I can check using ldd:

ldd: warning: you do not have execution permission for `./'
./ /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/ version `' not found (required by ./ =>  (0x00007fff47518000) => /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/ (0x00007f4e3fe41000) => /usr/lib64/ (0x00007f4e3fb3a000) => /usr/lib64/ (0x00007f4e3f838000) => /usr/lib64/ (0x00007f4e3f622000) => /usr/lib64/ (0x00007f4e3f406000) => /usr/lib64/ (0x00007f4e3f038000)
	/lib64/ (0x00007f4e4036d000) => /usr/lib64/ (0x00007f4e3ee34000) => /usr/lib64/ (0x00007f4e3ec2c000)

So the crucial line

./ /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/ version `' not found (required by ./

Tells me that the shared object can load the correct version. I get the same error when importing the module in python.