Correct way to do backpropagation through time?

[DuaneNielsen]

Check out hooks. If you want to inspect an gradient, you can register a backwards_hook, and drop the values into a print statement or tensorboard.

Why cant I see .grad of an intermediate variable?

Hi Kalamaya, By default, gradients are only retained for leaf variables. non-leaf variables’ gradients are not retained to be inspected later. This was done by design, to save memory. However, you can inspect and extract the gradients of the intermediate variables via hooks. You can register a function on a Variable that will be called when the backward of the variable is being processed. More documentation on hooks is here: http://pytorch.org/docs/autograd.html#torch.autograd.Variable.regis…

eg, in the below code I drop a hook to monitor the values passing through a softmax functiion. (later I compute the entropy and pump it into tensorboard).

        def monitorAttention(self, input, output):
            if writer.global_step % 10 == 0:
                monitors.monitorSoftmax(self, input, output, ' input ', writer, dim=1)
        self.softmax.register_forward_hook(monitorAttention)