Newbie question: Grad set to None in second optimizer pass

I am implementing a manual gradient descend to understand gradients better.

class Optimizer:
    def __init__(self, m, b, learning_rate):
        self.m = m
        self.b = b
        self.learning_rate = learning_rate
        print (self.m, self.b)
    def calculate_error(self, x_data, y_label):        
        error = 0
        for x, y in zip(x_data, y_label):
            y_hat = self.m * x + self.b
            error = error + ((y_hat - y)**2)
        return error

    def optimize(self):
        loss = self.calculate_error(x_data_source, y_label_source)
        print (self.m.grad, self.b.grad)
        self.m = self.m - self.m.grad * self.learning_rate
        self.b = self.b - self.b.grad * self.learning_rate
        print (self.m.requires_grad)
        return loss

Running the Optimizer once, results in expected results, the second time I get the error:

---> 21         self.m = self.m - self.m.grad * self.learning_rate
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'

The console output of the print statements indicate the requires grad stays True, but stll it is not recalculated

tensor([-727.3233]) tensor([-94.7521])
None None

It look like right after the recalculation of the variables (m and b) their grad is reset to None

What am I missing here

Try to change your weight update code to:

with torch.no_grad():
    self.m -= self.m.grad * self.learning_rate
    self.b -= self.b.grad * self.learning_rate

We would like to mutate the values in-place without building a new computation graph, so you should use the torch.no_grad context manager.

Alternatively, you could update the underlying .data.
It’s usually not recommended, but would also work: -= self.m.grad * self.learning_rate -= self.b.grad * self.learning_rate

Nice, thanks! It works now.
For any interested:
I tried the in place variant too, but without the no_grad context. That gave me an error that in place is not allowed.
And by the way, I had to zero the gradient after each step:

Anyone have any idea why the context manager is called no_grad(). It sounds like no gradients are begin computed, but what I understand from you it makes sure the graph is not recalculated.
I find the naming in ML libraries I encountered pretty cryptic at times, but that might be caused by me being new to this :slight_smile: