Spectral Norm in Pytorch

Hi,

Do we have Spectral Norm based regularizer in pytorch? Similar to what we have in case of l2 loss.
Thanks!

You can use torch.svd() and find the largest singular value (it automatically sorts the singular values).

However, this would be quite inefficient, because unfortunately pytorch does not currently support top-1 eigenvalue decomposition. See my recent question here.

I was wondering if we had Power Iteration way of doing it !

But If I do use torch.svd(), will it be auto diffrentiable , I mean, If I add this as a regularizer (ex: in place of l2 loss), Since calculating max singular value, through this way would not be diffrentiable. And we need it to be diffrentiable, since this would be used in calculating loss and updating.

Power iteration would be definitely great. But right now it is not available and I am not sure if it is in the plan.

Wouldn’t this be an issue:
But If I do use torch.svd(), will it be auto diffrentiable , I mean, If I add this as a regularizer (ex: in place of l2 loss), Since calculating max singular value, through this way would not be diffrentiable. And we need it to be diffrentiable, since this would be used in calculating loss and updating.

Yes, it is differentiable. I have used it.

1 Like