Hi, I’m trying to understand where / how to find the CUDA implementations for various operators. Specifically, the _pad_enum
op right now. In my code I call:
auto options =
F::PadFuncOptions({/*left*/ 0, /*right*/ 0, /*top*/ 0, /*bottom*/ pad_prob_m})
.mode(torch::kConstant)
.value(0);
F::pad(tensor, options);
which calls at::_pad_enum(). In my version of libtorch (libtorch-cxx11-abi-shared-with-deps-1.13.1+cu117.zip
), that implementation is in libtorch/include/ATen/ops/_pad_enum.h
and calls at::_ops::_pad_enum::call(self, pad, mode, value)
. If I look at at::_ops::_pad_enum_ops.h
I see this struct:
struct TORCH_API _pad_enum {
using schema = at::Tensor (const at::Tensor &, at::IntArrayRef, int64_t, c10::optional<double>);
using ptr_schema = schema*;
// See Note [static constexpr char* members for windows NVCC]
STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(name, "aten::_pad_enum")
STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(overload_name, "")
STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(schema_str, "_pad_enum(Tensor self, int[] pad, int mode, float? value=None) -> Tensor")
static at::Tensor call(const at::Tensor & self, at::IntArrayRef pad, int64_t mode, c10::optional<double> value);
static at::Tensor redispatch(c10::DispatchKeySet dispatchKeySet, const at::Tensor & self, at::IntArrayRef pad, int64_t mode, c10::optional<double> value);
};
So my questions are:
- Where is the underlying implementation here so I can look at the CUDA code?
- If I set a breakpoint there with cuda-gdb, and build from source with Debug build type, will it build with nvcc -g -G and will I be able to hit my breakpoint? (I assume I can’t step into from, e.g.
torch::nn::functional
because of this dispatch mechanism) - Is there any documentation describing how things like these include files are generated and how these calls are dispatched so I can understand this all better and be able to find any CUDA code I want?
Thanks,
-Collin