Parameters with zero .grad change in value - how to exclude some parameters from backpropagation update

[tom (Thomas V)]

@venkatasg

The gradient is zero, however the value of both the parameters(in this case, the weights and biases of the Linear modules) change every minibatch - only one of them should change.

You’re not using an optimizer with momentum by chance?
For those, the momentum will cause updates even when the gradients are zero.
(It’s also doing funny things to the statistics, probably, but with dropout we rarely think about it too much.)

As a trick you can (at least you could last time I checked) set the gradients to “None” instead of just zeroing them and then the parameters won’t be updated.

A side question - Is there a good way to visualize the history a variable while debugging in python? Preferably not inside a jupyter notebook.

Some people like TensorboardX.

Best regards

Thomas