I have just started learning this awesome tool called PyTorch but sadly I am stuck in an equivocal situation.
Below is a code snippet from one of the tutorials
with torch.no_grad():
weights -= weights.grad * lr
bias -= bias.grad * lr
weights.grad.zero_()
bias.grad.zero_()
I am kind of confused that even if I will do parameter update without using torch.no_grad() ( i.e. only in-place ) and since the backward call has already been made in the code above this snippet( not included in the snippet above ) which basically means all the âgradâ attributes are already computed and donât require the âoriginalâ values again. Then, why is it illegal to do those operations without torch.no_grad()?
I know it will flag off the error in PyTorch but I just wanted to know where my line of thought is at fault?
It is illegal because it would mean that the weight update would be done in a differentiable manner.
So that means that in the next iteration, when computing the gradients, it will flow back through multiple iterations of your training loop which is most likely not what you want to do !
Will you please clear where am I wrong in the below thought-process?
Assume, I only have one trainable weight parameter to update. So, on calling loss.backward() the gradient of loss wrt that weight has been saved in the node corresponding to the tensor comprising of that weight.
The way I perceive in-place operation is:- "The new value of weight oust the previous value in the âsameâ node i.e. according to me( unsure if itâs correct ), there is no change in the computational graph other than the âvalueâ update or no new node/edge has been added.
Going by my perception, since except values nothing else has changed, it shouldnât be illegal.