Following the tutorial from Peter Goldsborough, I’ve implemented a CPP and CUDA extension to PyTorch. However, according the tutorial compiles each as a separate module. In practice, it seems more useful to have a single module that enables both operations. I’ve split up my CUDA and CPP extensions, however, into separate files. While it compiles and works just fine in a pair of CPP and CUDA files like in the tutorial, it now fails miserably. All I’ve done is to separate the declarations into header files, the CPP code into its own file, and the CUDA code into its own file.
Right now, my setup.py looks like:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension, CppExtension
setup(
name='test',
ext_modules=[
CUDAExtension(
name='test',
sources=['bindings.cpp',
'test_cuda.cu',
'other_test_cuda.cu',
'test.cpp',
'other_test.cpp'],
include_dirs=['./include']
),
],
cmdclass={
'build_ext': BuildExtension
})
and jit.py is:
from torch.utils.cpp_extension import load
test = load(
name='test',
sources=['bindings.cpp',
'test.cpp',
'test_cuda.cu',
'other_test.cpp',
'other_test_cuda.cu'],
extra_include_paths=['./include']
verbose=False)
help(test)
The sources are all my implementations, while I have the declarations (four files: test.h, test.cuh, other_test.h, and other_test.cuh) in the ./include directory.
I moved the PYBIND11_MODULE to bindings.cpp as suggested by this StackOverflow answer. That file looks like this:
#include <torch/extension.h>
#include "test.h"
#include "other_test.h"
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
m.def("test_a", &test_a, "Test A");
m.def("test_b", &test_b, "Test B");
m.def("other_test_a", &other_test_a, "Other Test A");
m.def("other_test_b", &other_test_b, "Other Test B");
}
My files are laid out so that I have
-include
-test.h
-other_test.h
-test.cuh
-other_test.cuh
-test.cpp
-test.cu
-other_test.cpp
-other_test.cu
-bindings.cpp
-jit.py
-setup.py
This all fails terribly when I try to build. It appears to be a PyBind11 issue, but because of PyTorch’s abstraction, it’s not obvious how to solve it looking through PyBind only. A sample of the error printout is below
python setup.py build
running build
running build_ext
building 'test' extension
/usr/local/cuda/bin/nvcc -I./include -I...torch/lib/include -I...torch/lib/include/torch/csrc/api/include -I...torch/lib/include/TH -I...torch/lib/include/THC -I/usr/local/cuda/include -I/data/anaconda3/envs/test/include/python3.7m -c test_cuda.cu -o build/temp.linux-x86_64-3.7/test_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=other_test -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
...torch/lib/include/ATen/core/jit_type.h(945): warning: statement is unreachable
...torch/lib/include/pybind11/pytypes.h:1165:206: error: expansion pattern ‘pybind11::detail::negation<std::is_same<pybind11::detail::bools<pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ..., true>, pybind11::detail::bools<true, pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ...> > >::value’ contains no argument packs
...torch/lib/include/pybind11/pytypes.h:1165:215: error: template argument 1 is invalid
...torch/lib/include/pybind11/pytypes.h:1165:392: error: expansion pattern ‘pybind11::detail::negation<std::is_same<pybind11::detail::bools<pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ..., true>, pybind11::detail::bools<true, pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ...> > >::value’ contains no argument packs
...torch/lib/include/pybind11/pytypes.h:1165:395: error: template argument 2 is invalid
...torch/lib/include/pybind11/pytypes.h:1165:397: error: template argument 1 is invalid
...torch/lib/include/pybind11/pytypes.h:1165:397: error: template argument 2 is invalid
...torch/lib/include/pybind11/pytypes.h:1165:412: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:766:149: error: expansion pattern ‘std::is_copy_constructible<_Tp>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:766:240: error: expansion pattern ‘std::is_same<typename Container::value_type&, typename Container::reference>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:766:249: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:766:249: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:766:312: error: expansion pattern ‘std::is_copy_constructible<_Tp>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:766:403: error: expansion pattern ‘std::is_same<typename Container::value_type&, typename Container::reference>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:766:406: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:766:406: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:766:408: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:766:408: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:766:423: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:766:425: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:960:126: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:960:170: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char16_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:960:214: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char32_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:960:257: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, wchar_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:960:260: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:960:260: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:960:260: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:960:260: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:960:262: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1250:138: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1250:186: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char16_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1250:234: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char32_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1250:281: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, wchar_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1250:284: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1250:284: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1250:284: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1250:284: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1250:286: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1250:301: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1250:303: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1261:121: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1261:169: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char16_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1261:217: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, char32_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1261:264: error: expansion pattern ‘pybind11::detail::negation<std::is_same<CharT, wchar_t> >’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1261:267: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1261:267: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1261:267: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1261:267: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1261:269: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1261:284: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1261:286: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1583:112: error: expansion pattern ‘std::is_void<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:137: error: expansion pattern ‘std::is_pointer<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:164: error: expansion pattern ‘std::is_reference<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:187: error: expansion pattern ‘std::is_const<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:190: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:190: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1583:190: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1583:190: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1583:192: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:201: error: expected parameter pack before ‘...’
...torch/lib/include/pybind11/cast.h:1583:251: error: expansion pattern ‘pybind11::detail::negation<pybind11::detail::is_copy_constructible<T1> >::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:294: error: expansion pattern ‘std::is_move_constructible<_Tp>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:392: error: expansion pattern ‘std::is_same<decltype (declval<pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type, void> >().operator T&()), T&>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:401: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:401: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1583:401: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1583:401: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1583:453: error: expansion pattern ‘std::is_void<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:478: error: expansion pattern ‘std::is_pointer<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:505: error: expansion pattern ‘std::is_reference<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:528: error: expansion pattern ‘std::is_const<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:531: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:531: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1583:531: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1583:531: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1583:533: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:542: error: expected parameter pack before ‘...’
...torch/lib/include/pybind11/cast.h:1583:592: error: expansion pattern ‘pybind11::detail::negation<pybind11::detail::is_copy_constructible<T1> >::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:635: error: expansion pattern ‘std::is_move_constructible<_Tp>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:733: error: expansion pattern ‘std::is_same<decltype (declval<pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type, void> >().operator T&()), T&>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1583:736: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1583:736: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1583:736: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1583:736: error: template argument 5 is invalid
...torch/lib/include/pybind11/cast.h:1583:738: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:738: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1583:753: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1583:755: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1590:121: error: expansion pattern ‘std::is_void<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:146: error: expansion pattern ‘std::is_pointer<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:173: error: expansion pattern ‘std::is_reference<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:196: error: expansion pattern ‘std::is_const<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:199: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:199: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1590:199: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1590:199: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1590:201: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:210: error: expected parameter pack before ‘...’
...torch/lib/include/pybind11/cast.h:1590:250: error: expansion pattern ‘pybind11::detail::negation<pybind11::detail::move_always<T> >::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:293: error: expansion pattern ‘std::is_move_constructible<_Tp>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:391: error: expansion pattern ‘std::is_same<decltype (declval<pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type, void> >().operator T&()), T&>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:400: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:400: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1590:400: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1590:400: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1590:452: error: expansion pattern ‘std::is_void<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:477: error: expansion pattern ‘std::is_pointer<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:504: error: expansion pattern ‘std::is_reference<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:527: error: expansion pattern ‘std::is_const<_Tp>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:530: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:530: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1590:530: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1590:530: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1590:532: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:541: error: expected parameter pack before ‘...’
...torch/lib/include/pybind11/cast.h:1590:581: error: expansion pattern ‘pybind11::detail::negation<pybind11::detail::move_always<T> >::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:624: error: expansion pattern ‘std::is_move_constructible<_Tp>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:722: error: expansion pattern ‘std::is_same<decltype (declval<pybind11::detail::type_caster<typename pybind11::detail::intrinsic_type<T>::type, void> >().operator T&()), T&>::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1590:725: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1590:725: error: template argument 3 is invalid
...torch/lib/include/pybind11/cast.h:1590:725: error: template argument 4 is invalid
...torch/lib/include/pybind11/cast.h:1590:725: error: template argument 5 is invalid
...torch/lib/include/pybind11/cast.h:1590:727: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:727: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1590:742: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1590:744: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1670:100: error: expansion pattern ‘pybind11::detail::move_always<T>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1670:138: error: expansion pattern ‘pybind11::detail::move_if_unreferenced<T>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1670:141: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1670:141: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1670:143: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1670:155: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1700:99: error: expansion pattern ‘pybind11::detail::move_always<T>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1700:137: error: expansion pattern ‘pybind11::detail::move_if_unreferenced<T>’ contains no argument packs
...torch/lib/include/pybind11/cast.h:1700:140: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1700:140: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:1700:142: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:1700:154: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:2074:145: error: expansion pattern ‘pybind11::detail::negation<pybind11::detail::negation<std::is_same<pybind11::detail::bools<pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::args_proxy, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ..., true>, pybind11::detail::bools<true, pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::args_proxy, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ...> > > >::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:2074:154: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:2074:278: error: expansion pattern ‘pybind11::detail::negation<pybind11::detail::negation<std::is_same<pybind11::detail::bools<pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::args_proxy, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ..., true>, pybind11::detail::bools<true, pybind11::detail::negation<std::is_base_of<pybind11::arg, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::args_proxy, Args> >::value ..., pybind11::detail::negation<std::is_same<pybind11::detail::kwargs_proxy, Args> >::value ...> > > >::value’ contains no argument packs
...torch/lib/include/pybind11/cast.h:2074:281: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:2074:283: error: template argument 1 is invalid
...torch/lib/include/pybind11/cast.h:2074:283: error: template argument 2 is invalid
...torch/lib/include/pybind11/cast.h:2074:298: error: template argument 1 is invalid
...
This goes on for hundreds of more lines.
Does anyone know how to build a PyTorch module extension where the module is broken up into more than two files?
