Saw an interesting issue. We have a custom C++ op in our code that works with PyTorch 0.4.1 but was throwing a pybind11 type mismatch exception with PyTorch 1.0.1. After a bunch of debugging, I figured out that PyTorch pip package contains pybind11 header files which are being used when compiling the torch c++ custom op (using the setup tools option).
Unfortunately we have other library code (outside of pytorch) that is also pybind11 bound. Lets say this has a object foo in here. For our use case, we need to create object foo in python and pass the reference to it to the Pytorch c++ custom op. Essentially for this use case, we need to match the pybind11 versions between what this foo code uses and what PyTorch uses.
So my questions/observations
I could not find any reference in the documentation that suggested that PyTorch releases ship with a certain version of pybind11 headers. So this surprised me.
How do I know which specific version of pybind11 headers are contained in a given PyTorch release? The header file code itself does not seem to indicate version. Specifically I need this information for 1.0.1 release
Note that foo code in the above use case was using pybind11 that shipped with Ubuntu 16.04 release. So this likely makes PyTorch 1.0.1 incompatible for Ubuntu 16.04. Even though I admit the use case I mentioned must be exceedingly rare for PyTorch users.
This topic is old but I just sliced my toes off with a similar issue at head and thought I’d document what I found. I had a similar use case (pybind11 method in a project that depends on PyTorch needing to accept a torch::jit::StrongFunctionPtr and being somewhat unsuccessful at it in certain situations).
Recent PyTorch wheels do include pybind11 headers corresponding to the submodule, afaict, and I was using a workaround in my build setup to always include PyTorch headers at the front of the include path for the parts of the project that depend on pybind11. Not great, but worked, and I can’t think of a better way to depend on a binary distribution like that.
However, when I built from source, my installed package did not include the headers. I believe this is because of detection logic choosing to use the version of pybind11 that I pip installed (i.e. pip install pybind11). In my situation, I also had a system package for pybind installed at an old version. I spent several hours untangling which were used where and systematizing it.
It would be really handy if both PyTorch and pybind11 stamped the actual version or commit hash used somewhere in their distribution. Maybe even a symbol in the resultant binaries that carried the version in its name somehow (so it could be inspected with nm). In the absence of that, some better documentation of how this dependency is resolved would help a lot.
It could also work if we all just uniformly switched to the pip installed versions and included the right version spec in our requirements.txt file. That would probably break a lot of people, though, who don’t really need cross-module coordination on types.