Contractive Loss rising in CAE - Contractive Autoencoder

#1

Hello,
I am builing a Contractive Autoencoder. My MSELoss is decreasing, but unfortunately my Contraction Loss is rising, wheres it should be different.

Did anyone encounter similar issues?

Here is my Loss Function for the Contractive Loss: I use LeakyReLU as activation and MSE.

def loss_function(W, x, predicted, h, lam):
mse = loss_crit(predicted, x)

dh = torch.where(h >= 0 , torch.ones(1), torch.ones(1)*1e-2) 

w_sum = torch.sum(W**2, dim=1)

w_sum = w_sum.unsqueeze(1)

contractive_loss = torch.sum(torch.mm(dh**2,w_sum), 0)

return mse + contractive_loss * lam, mse, contractive_loss