In my experiment setting, I am trying to do some data pre-processing using C++ code. I have tried two approaches:
Using subprocess to run C++ built executable and save the results to the hard disk. Then on the Python side, it reads the processed result.
Using pybind11 to warp the C++ code, and return a buffer to Python directly, which can be converted to numpy array.
Both methods have been tested to be correct and coherent solely. However, when being integrated inside a dataloader (worker number > 1), the second method met some problem. The situation was that the training would get some NaN value very soon. I do not know how to look into this. Is there any suggestion and tip for me to use pybind11 warpped C++ code along with pytorch dataloader?
Do you use the builtin cpp_extension to bind your cpp code? That would allow you to work directly with Tensors and avoid potential errors where your buffer memory might get freed/reused before your python code is done with it.
No I do not use that. But I do return a buffer cv::Mat buffer, which is accessed by py::bufer_info and is converted to numpy array then. This numpy array is then converted to numpy array by torch.from_numpy.
Other information is that I was building the python module using pybind11 2.4 with C++ standard 11. I have seen some related question on pybind11 thread that talked about potential memory leak using C++ standard 11 building pybind11 2.4.