Hi,
I want to call some backward functions like:
AddBackward
MulBackward
FftWithSizeBackward
etc…
Where are they and how can I call them with my input?
Thanx
It depends on what you used the loss function.
It does not depends on loss function, I just want to call the function AddBackward for example. This call is individual and has nothing to do with neural network! @fmassa @jekbradbury
I mean, for example, whatever you used the loss function like nn.CrossEntropyLoss, it can be NllLossBackward …
what you want is just call the backward function manually, check this documentation. (https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html).
As I said please forget about loss! For each operation we have a forward function which we can call as easy as calling for ex: torch.add(x,y). And we have backward function which I dont know how to call manually for example something like torch. AddBackward(ctx,x,y)! I know that autograd call them one by one somehow but I dont know how to call it my self!
Hi,
Can you please avoid pinging everyone? It creates a lot of noise for us and we do read all the posts.
It depends on which functions you want? There is no common api to access the backward functions without doing the forward.
Thanks for your reply. Sure I wont do it in future.
I want to call these for example:
AddBackward
MulBackward
FftWithSizeBackward
I don’t mind calling them in c++ or python. I just want to see the definition and call them later on.
For the forward function doing o = x + y, the backward is gx = go and gy = go.
For the forward function doing o = x * y, the backward is gx = y * go and gy = x * go.
For the fft, it depends on which forward function you use. If you use a regular o = fft(x), I think the gradient is just gx = ifft(go). If you want to be sure, you can find here how the top level fft functions are linked to the low level one called _fft_with_size. Here you can find how the backward of that function is defined for each of it’s arguments. And the fft_backward() can be found here. Unfortunately thiss function is not accessible directly as the arguments are quite far from the regular python api. But you can easily find the corresponding torch.[]fft call that correspond to your forward.
2 Likes
Thnx 
Another follow up question, what is grad[0] and grad[1] and .defined()?
Tensor var_std_mean_backward(const variable_list& grads, const Tensor & self, const Tensor & r1, const Tensor & r2, bool unbiased, bool is_std) {
Tensor grad;
if (grads[0].defined()) {
grad = is_std ? std_backward(r1, grads[0], self, unbiased) : var_backward(grads[0], self, unbiased);
}
if (grads[1].defined()) {
Tensor mean_grad = mean_backward(grads[1], self.sizes(), self.numel());
grad = grads[0].defined() ? grad + mean_grad : mean_grad;
}
return grad;
}
And where are these grad_fn coming from
<AddBackward0 at 0x1185d8048>
?
a=torch.rand(2,3).requires_grad_()
b=torch.rand(2,3)
z=(a+b).requires_grad_()
f=z.sum()
f.backward()
z.grad_fn
Out[17]: <AddBackward0 at 0x1185d8048>
surpassingly if you search whole repository for the AddBackward you dont find anything. So where are these string coming from?
https://github.com/pytorch/pytorch/search?q=AddBackward&unscoped_q=AddBackward
grad[0] and grad[1] are the gradient wrt the 0th and 1st output of the forward function. What I called go for my add and backward examples above.
You can’t find these functions because they are automatically generated.
The gradients for most ops are defined in the derivatives.yaml file. And the corresponding cpp binding is generated automatically.
2 Likes