What is the difference between torch.relu, torch.nn.ReLU and torch.nn.functional.relu?

[Nikronic (Nikan Doosti)]

Hi,

You may want to know about some tips here:

What's difference of nn.Softmax(), nn.softmax(), nn.functional.softmax()?

Hi. I guess there are three way to use activation function in a custom module: Use nn.Softmax() in the initializer in the custom model. Use nn.softmax() in the forward function in the custom model. Use nn.functional.softmax() in the forward function in the custom model. What are the differences among them and what is the proper way to use an activation function in my custom model?

Why there isn't a method named torch.softmax

So the idea is to put more deep-learning-oriented functions in torch.nn.functional and keep general-purpose functions in under torch directly. softmax was deemed to fall into the former, sigmoid in the latter category. While there is torch.softmax, this is by accident (which is why it is not documented) rather than as a design (previous version of PyTorch didn’t have the fancy torch._C._nn module to put the C++-implementations of torch.nn.functional-functions). I would advise to only use the do…

Bests