I was reading about different implementations of the ReLU activation function in Pytorch, and I discovered that there are three different ReLU functions in Pytorch. As I read this post, I realized that the difference between torch.nn.ReLU
and torch.nn.functional.relu
is more about the coding style. However, there is a third function, torch.relu
, which has the same functionality as torch.nn.functional.relu
. Considering this, my main question here is that what is the difference between torch.relu
and torch.nn.functional.relu
, and can we use them interchangeably?
1 Like
Hi,
The short answer is none.
The longer answer is that our binding code to cpp is set up so that most low level optimized functions (like relu) get bound to the torch.foo
namespace.
In this case, you can use torch.relu
and torch.nn.functional.relu
interchangeably yes.
4 Likes
Hi,
You may want to know about some tips here:
Bests
1 Like