I’m trying to figure out how the python bindings are generated as I see that pybind is used in some cases but it also seems like some python bindings are generated by custom files: https://github.com/pytorch/pytorch/blob/master/tools/setup_helpers/generate_code.py
Understanding how and why this works will be super helpful to me as I am trying to create C# bindings for libtorch.
Most of the core PyTorch functions on the
torch package and on
torch.Tensor are generated from Declarations.yaml (*) using gen_python_functions.cpp. The generated files look like:
We don’t use pybind11 here for a few reasons:
- We need to disambiguate multiple overloads in a PyTorch-specific way. Some of this behavior is due to historical Lua Torch function signatures. We try to avoid this ambiguity in new functions.
- pybind11 tends to slow down compile time a lot
(*) Declarations.yaml is generated from native_functions.yaml. There used to be other inputs that went into Declarations.yaml, but I’m not sure if that’s still the case. Anyways, Declarations.yaml is intended as the “public” machine-readable API specification for things like language bindings.
Because the bindings I am trying to make are in C# which supports method overloading, and compile time is not one of my constraints, would using a 3rd party bindings generator (similar to pybind) be recommended or are there other considerations?
I don’t understand. How would pybind help with C#? Aren’t you trying to bind C++ functions in C#? (pybind11 binds C++ functions to Python)
I would not be using pybind I would be using a generator that would be for C# similar to the way pybind is for python, something like http://www.swig.org/tutorial.html
I see that Declarations.yaml is generated when building pytorch. Ideally, when auto-generating bindings for libtorch, I wouldn’t have to run the complete build. Instead, I’d like to generate the Declarations.yaml file and then download the pre-built packages from Start Locally | PyTorch. Is it possible for me to only run the build step that generates Declarations.yaml given a download of pytorch source?
It seems that running
python -m tools.codegen.gen -s aten/src/ATen -d torch/share/ATen from root of a source checkout puts Declarations.yaml in the expected place, though I’m not sure if it inappropriately puts other stuff in that folder as well.