Caculating Hessian matrix and eigenvalues

1

Hi,

I am trying to calculate Hessian matrix and eigenvalues/vectors for below neural network:

dataset = TensorDataset(x,y)

dataloader = DataLoader(dataset, batch_size=batchsize,shuffle=True,num_workers = 0,drop_last = False )
class Net2_u(nn.Module):

    #The __init__ function stack the layers of the 
    #network Sequentially 
    def __init__(self):
        super(Net2_u, self).__init__()
        self.main = nn.Sequential(
            nn.Flatten(),
            nn.Linear(input_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            nn.Linear(h_n,h_n),
            
            Swish(),
            
            nn.Linear(h_n,1),
        )
    def forward(self,x):
        output = self.main(x)
        return  output

net2_u = Net2_u().to(device)
optimizer_u = optim.Adam(net2_u.parameters(), lr=learning_rate, betas = (0.9,0.99),eps = 10**-15)

for inputs, targets in dataloader:
break
inputs, targets = inputs.cuda(), targets.cuda()

for epoch in range (epochs):
for batch_idx, (x_in,y_in) in enumerate(dataloader):
net2_u.zero_grad()
loss = criterion(x,y)
loss.backward()
optimizer_u.step()
hessian_comp = hessian(net2_u, loss, data= (inputs, targets), cuda=True)
top_eigenvalues, top_eigenvector = hessian_comp.eigenvalues()

I am getting this error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (256x1 and 2x70)

What is this error for?
Can someone tell me how i am able to calculate the Hessian matrix for this kind of network?

2

torch.autograd.functional.hessian doesn’t accept a data argument so I guess you are using a custom implementation?
Does your code work without this custom implementation or with the built-in one?

4

I imported hessian from PyHessian library.

Without this two lines, hessian_comp = hessian(net2_u, loss, data= (inputs, targets), cuda=True)
top_eigenvalues, top_eigenvector = hessian_comp.eigenvalues(), my code works very well but when i want to calculate hessian matrix and add hessian line it gives the error i mentioned.

I tried torch.autograd.functional.hessian a few weeks ago but i can remember that i got error for this one too.