How does pytorch use c++?

Hey there,

I started to use pytorch two days ago and I was trying to figure out how it all works.

I was trying to see where the implementation of ReLU is, but, after a certain point of backtracking in python code, I did not find the implementation.

Then I check on the github and found the implementation, but where exactly is that binary on linux? Couldn’t find it. And how/where is the python code calling it?

Thank you.

Slightly out of date but still quite useful blog post: http://blog.ezyang.com/2019/05/pytorch-internals/

I’m still learning myself but my understanding is that PyTorch uses a dispatcher mechanism to select an appropriate c++ implementation of various operators. This is covered in detail in another great blog post from Ed: http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/

Since PyTorch operators work on various hardware platforms, tensor types, and data types - the underlying operator implementation can be optimized in C++ by humans, codegen, and compilers.

For relu in particular it looks like the c++ version just calls threshold:

Here’s the CPU implementation for threshold:

Hope that helps!

2 Likes