How to calculate the Jacobian matrix of a neural network?

1

How to calculate the Jacobian matrix of a neural network?

2

Have you tried torch.autograd.functional.jacobian?

3

Yes,I tried, but it seems run as I get the gradient all zero

4

And my network Is just a single fullly connected network, no non-linear activation function, so I think the gradient should not change when the input change. It seems the gradient is changing as the input change.

5

Could you share a small snippet that shows this? It seems to not give all zeros for me:

net = nn.Sequential(
    nn.Conv2d(3, 16, kernel_size=3, padding=1),
    nn.AvgPool2d(2),
    nn.Flatten(1),
    nn.Linear(16*4*4//4, 10)
)
input = torch.randn(1, 3, 4, 4)
torch.autograd.functional.jacobian(net, input)

tensor([[[[[[-6.3562e-03, -1.0175e-02, -2.5312e-02, -1.1178e-02],
            [-1.4468e-02,  1.4982e-04, -7.6495e-03, -8.9791e-04],
            [-1.8474e-02, -2.5984e-02, -2.6214e-02,  8.7420e-03],
            [-6.6065e-03, -1.8409e-02, -6.9709e-03,  1.7825e-02]],
            ...
6

http://211.81.52.14:8888/notebooks/jupyter/Untitled2.ipynb
hello,sir, can you open this link

7

It won’t load. Try a Google Colab.

8

hello,sir, This one

9

I don’t think you’re passing the right inputs to it.

Instead of this:

func = lambda *params: model(data)
jacs = F.jacobian(func, params)

It should be this:

jacs = F.jacobian(model, data)
10

I am confused why the different input get different gradient, as you know, The network is a single fully-connected neural network. y=W*x+b, so when calculate dy/dx, we are supposed to get W. W should not change when x is changing.

11

The gradient of the network means dy/dw, not dy/dx.

12

so how to calculate the dy/dx

13

I found these threads that might help you:

14

well, Thank you very much