Building wheels with pytorch extensions

I am trying to build a manylinux wheel containing pytorch extensions using CUDA. Everything works fine except for patchelf’ing libcudart in the so files. After running auditwheel --repair on the wheel, the compiled extensions have been patched as I can check using ldd:

ldd nms_cuda.cpython-37m-x86_64-linux-gnu.so
ldd: warning: you do not have execution permission for `./nms_cuda.cpython-37m-x86_64-linux-gnu.so'
./nms_cuda.cpython-37m-x86_64-linux-gnu.so: /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/libcudart-1b201d85.so.10.1.243: version `libcudart.so.10.1' not found (required by ./nms_cuda.cpython-37m-x86_64-linux-gnu.so)
	linux-vdso.so.1 =>  (0x00007fff47518000)
	libcudart-1b201d85.so.10.1.243 => /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/libcudart-1b201d85.so.10.1.243 (0x00007f4e3fe41000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f4e3fb3a000)
	libm.so.6 => /usr/lib64/libm.so.6 (0x00007f4e3f838000)
	libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f4e3f622000)
	libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f4e3f406000)
	libc.so.6 => /usr/lib64/libc.so.6 (0x00007f4e3f038000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4e4036d000)
	libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f4e3ee34000)
	librt.so.1 => /usr/lib64/librt.so.1 (0x00007f4e3ec2c000)

So the crucial line

./nms_cuda.cpython-37m-x86_64-linux-gnu.so: /workspace/wheelhouse/mmdet-1.0rc1+e2c4ea3/mmdet/ops/nms/./../../../mmdet.libs/libcudart-1b201d85.so.10.1.243: version `libcudart.so.10.1' not found (required by ./nms_cuda.cpython-37m-x86_64-linux-gnu.so)

Tells me that the shared object can load the correct version. I get the same error when importing the module in python.