What is the best way to check if parameters already in optimizer?

[Nikronic (Nikan Doosti)]

Hi,

The taken approach is complex and not necessary really.
The idea is that when you freeze a parameter using requires_grad=False, even you pass entire parameters to the optimizer, it will not consider them and literally just skip those parameters in the way that you have never provided them and this will be done by not accumulating gradients for that particular parameter which also leads to have None as the gradient.

Passing to the optimizers frozen parameters

The parameters with requires_grad = False won’t accumulate a gradient during the backward pass and their .grad attribute will stay None by default (if no previous gradient was calculated). The optimizer will skip all parameters with a None gradient as seen here. All parameters will accumulate gradients and the optimizer will only update the passed parameters. If you call optimizer.zero_grad() and don’t use model.zero_grad(), the “unused” parameters will continue to accumulate gradients. If…

Bests