Hello! I am trying to learn the alpha and beta in the following equation, but I am really having a hard time getting something that doesn’t throw nan during backward.
The code in question:
(
self.beta.abs()
.mul(log(step / (self.n_steps - 1) + 1e-5))
.exp()
.mul(-1)
.add(1)
.log()
.mul(self.alpha.abs())
.exp()
.mul(self.initial_lr.abs())
)
initial_lr, alpha, and beta are nn.Parameter.
Any suggestions?
I’d also welcome some other interpolation recommendations with a nice concave to convex shape between (0, eps) and (T, 0). Though the boundary conditions must be satisfied.