Using Python embedded interpreter via Pybind in C++ extension

1

Is it possible to use python embedded interpreter via pybind in a C++ extension? I want to invoke a python function in c++ but get an error using py::module::import

My code looks like:

#include "torch/extension.h"

namespace py = pybind11;

void SomeFunction() {
  py::object python_func = py::module::import("python").attr("func")
  python_func();
}

I get the error:

image
image2768×328 161 KB

Do I have to set pybind11::embed somewhere?

I am not able to import torch/include/pybind11/embed.h as this file cannot be found

2

Do you hold the GIL properly when you call these python constructs.
Also did you name your module “python”? Maybe use a name less likely to be used in other places.

3

I missed out on acquiring the GIL but now I get the same error on the call to pybind11::gil_scoped_acquire gil

Am I missing a header in order to use pybind or do I need to start the interpreter by calling py::scoped_interpreter guard{} in my extension?

I didn’t name the module “python” that was just an example with some names changed.

4

You mean it sefault when you try to acquire the gil? Is python properly initialized alresady?

5

Yes it segfaults on acquiring the GIL and python is properly initialized

6

This looks unrelated to pytorch. The whole stack trace points to pybind and cpython code…
Could you give more information on how you build this?

7

I built it using instructions from https://pytorch.org/tutorials/advanced/cpp_extension.html

But I was able to resolve this issue using the workaround in: https://github.com/pybind/pybind11/issues/1738

8

Ho interesting. It does look like an initialization problem then :confused: