Linking tensor.grad.data to nodes

1

Hi guys. I have:

    result = model(tensor)

    score_max_index = result.argmax()
    score_max = result[score_max_index]

    score_max.backward()

    data = tensor.grad.data.abs()

Data is

tensor([0.4954, 0.6926, 0.2542, 1.0316, 0.3206, 0.0126, 0.5587, 0.0943, 2.8202,
        1.4996, 0.6181, 1.6888, 0.4801, 2.1043, 0.5233, 1.4717, 0.5883, 1.8086,
        1.2205, 0.5946, 1.1455, 0.4592, 1.6773, 1.2097, 1.0795, 1.1406, 0.8416,
        1.0722, 1.1076, 0.1615, 0.9181, 1.2613, 1.4448, 1.4703, 0.4576, 1.1672,
        0.5309, 3.9590, 0.5291, 7.3970, 0.4947, 0.4488, 0.8801, 0.1289, 0.3571,
        0.1130, 0.9194, 0.4625, 0.2476, 1.4286, 0.3808])

My network is:

self.linear1 = torch.nn.Linear(_input_dimension, _hidden_dimension)
        self.activation = torch.nn.Tanh()
        self.dropout = torch.nn.Dropout()
        self.linear2 = torch.nn.Linear(_hidden_dimension, _hidden_dimension2)
        self.activation2 = torch.nn.Tanh()
        self.linear3 = torch.nn.Linear(_hidden_dimension2, _hidden_dimension3)
        self.activation3 = torch.nn.Tanh()
        self.linear4 = torch.nn.Linear(_hidden_dimension3, _output_dimension)

How can I determine which member of tensor.grad.data “belongs” to which node or edge in that network?

Thanks,
Ian

2

It belongs to tensor (if present)?
To get the model’s parameter’s grad, you could loop through for name, p in model.named_parameters() and use p.grad for each.
The .data should probably be avoided.

Best regards

Thomas

3

So p.grad in that instance will be the gradient for the tensor’s run through the network?

4

It is good to always say it as “the gradient of function with respect to argument to that function” to keep the mathematical nature of things happy, so in this case p.grad will be the gradient of the loss (from loss.backward() - actually the sum of all of them since the beginning or last time you used opt.zero_grad or so) with respect to p.

5

Many thanks Tom, that is exceptionally clear and useful.

Ian