Hello everyone,
I have the loss function:
1/2 * torch.norm(output.mT - output.mT@weight, ‘fro’)**2 + gamma * torch.sum(abs(weight)) + lamda/4*torch.trace((torch.square(output@output.mT - torch.eye(len(output)))))
In the above loss function there are 3 components and obviously the have values of different sizes. Now if I choose my parameter gamma and lambda any random numbers then the during the training the output of the model collapse to zero ( I posted earlier about this). Now if I choose lambda and gamma such that second part and the last part of the loss function become smaller after multiplied by lambda and gamma then the output does not collapse to zero anymore and the loss function also decreases. Now my concern I choose the parameter once I observe the values of the second and the last components of the loss function but what is the general technique to chose the parameters involves in the loss function so that loss function works?