Thanks for explaining!
For a very quick and high-level description:
-
Tensor operations are implemented in the C++ Tensor library called ATen, which also includes some old code from Torch time (we are rewriting those for PyTorch 1.0!). It’s important to note that ATen doesn’t have autograd functionality (at least not currently) and only provides a Tensor API and operators on Tensors of various types, e.g., one of them being
CPUFloatTenor
.Since there are many different Tensor types and different operators, a lot of codegen is applied here, which unfortunately makes the code harder to read for new contributors. But we are constantly improving upon it!
-
To connect ATen with Python, we have a lot of C++ code living under
torch/csrc
, implementing things likeautograd
,jit
,dtype
device
layout
objects, etc. In particular, forautograd
, we subclass each of the ATen tensor types with something calledVariableType
that implements functionalities to track computation graph (i.e., history), which theautograd
engine will use to back-prop through and compute gradients.Similar to ATen, we also use some codegen to generate the operator interface on
VariableTypes
, and more importantly generate the backward pass for each operator. The mapping between forward and backward functions are listed in a yaml file, with many backward functions implemented inFunctions.cpp
. Note that all operations inFunctions.cpp
are operating onVariable
s, i.e., Tensors withVariableType
, which means that if a function, e.g.my_op_backward
, is implemented using existing operations that supportautograd
, the double backward also works out of the box! (Of course one may also implement a custom double backward for efficiency reasons.) -
Finally, the connection between C++ land (i.e.,
torch/csrc
) and Python is mostly implemented with Python C API and pybind11.