Pybind - Pytorch Segfault?

I have a simple toy example made in pybind / c++ that compiles:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/numpy.h>
#include <stdexcept>

#include <torch/extension.h>

namespace
{

namespace py = pybind11;

torch::Tensor test2(){
    auto label_map = torch::zeros({100, 100}).to(torch::kFloat32);
    return label_map;
}

} // namespace

// The first argument needs to match the name of the *.so in the BUILD file.
PYBIND11_MODULE(tools_binding, m)
{
    m.doc() = "Tools functions";

    m.def("test2",
          &test2,
          "test2"
        );

}

However, when i invoke test2(), it segfaults with

Segmentation fault

If i dont return anything however:

void test2(){
    auto label_map = torch::zeros({100, 100}).to(torch::kFloat32);
}

Then it doesnt segfault. Whats going on?

I did a GDB and found the following:

Program received signal SIGSEGV, Segmentation fault.
0x00007fffef60d014 in THPVariable_NewWithVar(_typeobject*, at::Tensor) ()