How to get gradients of output of activation function with respect to its input?

Hi,

I am wondering if there is a way to get gradients of output of activation function with respect to input to the activation. It is needed in backpropagation, so I am sure pytorch does that. I can probably use input values and take derivative as derivative of activation functions are easy, but is there a way to get that from pytorch? Also, I wonder what happens if the input is exactly 0 and our activation function is Relu?

Thanks.

Hi,

You can use autograd for this.

gradients = torch.autograd.grad(outputs=activations, inputs=input,
                          grad_outputs=torch.ones(activations.size()).to(device),
                          create_graph=True, retain_graph=True, only_inputs=True)[0]

Related question here. I’d like to compute the gradient of activations wrt inputs for several layers inside the network. So far, I’ve built several intermediate models to compute the gradients of the network output wrt input. I’m sure it can done more efficiently using hooks, but I don’t know how.

Any help ?
Thanks.