Hi all, new pytorch user here.
I was trying to implement an Autoencoder with PyTorch and used the function torch.linalg.norm. Because the sparce autoencoder I wish to implement uses the Frobenius norm as a regularization term along with MSE and sparcity, using l2 and l1 norm respectively, I wanted to keep my code clean and understantable. Therefore I used the Frobenius norm like the following:
reg = torch.linalg.norm(W, ord='fro') ** 2
where w is a h*d matrix.
My model comes from torch.optim.Adam. After executing the method backward, I was geting NaNs all over the place after. In an attempt to debug, after inserting
torch.autograd.set_detect_anomaly(True)
I located the issue. After a long time trying to find other ways to implement the Frobenius norm to avoid NaNs, I just typed
reg = torch.linalg.norm(W) ** 2
and it worked! From the documentation of torch.linalg.norm, I see that
None(default): for matrices: Frobenius norm
How can this be explained? Feel free to ask for more of my code or any debugging output I can produce. I wish to know whether it is a bug or if my torch noviceness is above level 9000.