When pytorch builtin optimizer loss.backward() could calculate gradient and it is proven correct in its calculation , even though why we need to check derivative computation? Whats wrong in my following code to check derivative computation in back propagation? Or we check gradient computation when we implement gradient computation using our python code not using pytorch loss.backward()?
for epoch in range(num_epochs):
# forward pass (prediction) and loss
y_predicted = model(X)
loss = criterion(y_predicted,y)
# backward loss
# calculate the gradients
loss.backward()
# update weights
optimizer.step()
optimizer.zero_grad()
running_loss +=loss.item()
gradcheck and gradgradcheck can be used for custom operations as a test and is also used internally in unit tests.
While I’m sure the gradient calculations are correctly implemented, I’m also sure they would break without proper testing due to (various) changes in the code base. 
1 Like
@ptrblck thanks for giving insight , but in above code as i mentioned there is no any custom code all is code belongs to builtin pytorch , in which scenario it may break?
I’m not sure, if I understand the question correctly, but it could break, if a developer pushes a change to the code base, which could contain a mistake in any calculation, which is why tests are needed.
If you don’t write custom autograd.Functions or work on the backend directly, these methods might be uninteresting for you.
1 Like
@ptrblck thanks , could you please share a simple code for writing custom autograd functions?
Still i am not clear if i write following code framework to train neural network then how could it be break? Calculating gradients loss.backward and update weights optimizer.step are builtin in pytorch how come it will break? Are you talking about changes made by pytroch team within new package/release of pytorch?
for epoch in range(num_epochs):
# forward pass (prediction) and loss
y_predicted = model(X)
loss = criterion(y_predicted,y)
# backward loss
# calculate the gradients
loss.backward()
# update weights
optimizer.step()
optimizer.zero_grad()
running_loss +=loss.item()
This tutorial explains how to write custom autograd.Fucntions.
If you are “just using” built-in PyTorch methods, you are most likely not interested in testing utilities.
Yes, since PyTorch is an open source framework with a lot of active developers, things can accidentally break, which is why testing is needed.
1 Like
@ptrblck thanks much appreciated - can you please just share a small piece of example where gradient is being checked , further what need to measure to decide graident is correct or not?