I started to use pytorch two days ago and I was trying to figure out how it all works.
I was trying to see where the implementation of ReLU is, but, after a certain point of backtracking in python code, I did not find the implementation.
Then I check on the github and found the implementation, but where exactly is that binary on linux? Couldn’t find it. And how/where is the python code calling it?
Slightly out of date but still quite useful blog post: http://blog.ezyang.com/2019/05/pytorch-internals/
I’m still learning myself but my understanding is that PyTorch uses a dispatcher mechanism to select an appropriate c++ implementation of various operators. This is covered in detail in another great blog post from Ed: http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/
Since PyTorch operators work on various hardware platforms, tensor types, and data types - the underlying operator implementation can be optimized in C++ by humans, codegen, and compilers.
For relu in particular it looks like the c++ version just calls threshold:
Here’s the CPU implementation for threshold:
Hope that helps!