Whats the difference between nn.relu() vs F.relu()

[Manik_Hossain]

whats the difference between nn.relu() vs nn.functional.relu()
and if there no difference so why such duplication…?

[ptrblck]

nn.ReLU() creates an nn.Module which you can add e.g. to an nn.Sequential model.
nn.functional.relu on the other side is just the functional API call to the relu function, so that you can add it e.g. in your forward method yourself.

Generally speaking it might depend on your coding style if you prefer modules for the activations or the functional calls. Personally I prefer the module approach if the activation has an internal state, e.g. PReLU.

[wwiiiii (Jeong TaeYeong)]

One example could be the availability of hook registration as below.

How to register hook function for functional form?

I’m now trying to implement Guided backpropagation from https://arxiv.org/abs/1412.6806 , which alters the gradient calculation in ReLU layer. When given model is implemented in the way containing nn.ReLU() instance as its submodule, I can easily register backward hook function to its every submodule as below. model = resnet101(pretrained=True) def hook(module, grad_input, grad_output): if isinstance(module, nn.ReLU): # do something return changed_grad_input for i, v in mod…