Bounded spectral norm

So I’ve been reading the spectral norm gaussian process paper. They have this part where they normalize the spectral norm weights based on the denominator of the equation. I was wondering how this can be done in PyTorch? Currently the issue is the spec norm layer is applied to a module and I don’t think it returns the weights denominator and numerator and if it did I’d have to go layer by layer and check if sigma(Wl) is above the threshold then multiply that, but all this isn’t possible because you instantiate SN over a module.

2 Likes