How are python bindings created?

1

I’m trying to figure out how the python bindings are generated as I see that pybind is used in some cases but it also seems like some python bindings are generated by custom files: https://github.com/pytorch/pytorch/blob/master/tools/setup_helpers/generate_code.py

Understanding how and why this works will be super helpful to me as I am trying to create C# bindings for libtorch.

2

Most of the core PyTorch functions on the torch package and on torch.Tensor are generated from Declarations.yaml (*) using gen_python_functions.cpp. The generated files look like:

python_torch_functions.cpp
python_variable_methods.cpp

We don’t use pybind11 here for a few reasons:

  1. We need to disambiguate multiple overloads in a PyTorch-specific way. Some of this behavior is due to historical Lua Torch function signatures. We try to avoid this ambiguity in new functions.
  2. pybind11 tends to slow down compile time a lot

(*) Declarations.yaml is generated from native_functions.yaml. There used to be other inputs that went into Declarations.yaml, but I’m not sure if that’s still the case. Anyways, Declarations.yaml is intended as the “public” machine-readable API specification for things like language bindings.

5 Likes
4

Because the bindings I am trying to make are in C# which supports method overloading, and compile time is not one of my constraints, would using a 3rd party bindings generator (similar to pybind) be recommended or are there other considerations?

5

I don’t understand. How would pybind help with C#? Aren’t you trying to bind C++ functions in C#? (pybind11 binds C++ functions to Python)

6

I would not be using pybind I would be using a generator that would be for C# similar to the way pybind is for python, something like http://www.swig.org/tutorial.html

7

I see that Declarations.yaml is generated when building pytorch. Ideally, when auto-generating bindings for libtorch, I wouldn’t have to run the complete build. Instead, I’d like to generate the Declarations.yaml file and then download the pre-built packages from Start Locally | PyTorch. Is it possible for me to only run the build step that generates Declarations.yaml given a download of pytorch source?

Thanks!

8

It seems that running python -m tools.codegen.gen -s aten/src/ATen -d torch/share/ATen from root of a source checkout puts Declarations.yaml in the expected place, though I’m not sure if it inappropriately puts other stuff in that folder as well.