Using softplus to learn sigma


I have a model which fits a mean and sigma for some data.

I have parameterised sigma using softplus:

q_sigma = torch.nn.Parameter(torch.nn.functional.softplus(torch.randn(n, dim)))

but encounter this error because the q_sigma value become a small negative number (-0.0015)

  File "/home/vidhi/anaconda3/lib/python3.8/site-packages/torch/distributions/", line 53, in __init__
    raise ValueError("The parameter {} has invalid values".format(param))

ValueError: The parameter scale has invalid values

I’m guessing this is classic numerical instability issue, but is there a better transform to learn sigma?


Hi Vidhi!

It’s not clear to me what your are trying to do. But I will cheerfully

Because you make q_sigma a Parameter I assume that you are
training it with some optimizer.

You initialize it with a positive value (the result of softplus()).

Even though q_sigma started out positive, the optimizer has know way
of knowing that q_sigma is supposed to remain positive, so at some
point it takes an optimizer step that makes q_sigma negative.

I would suggest instead that you train a parameter that is allowed to
be negative, and then derive a positive q_sigma from it:

q_parameter = torch.nn.Parameter (torch.randn (n, dim))

Then later in your code when you need (the positive) q_sigma, simply
derive it from q_parameter (with something like softplus() that
guarantees a positive result).

The point is, instead of training q_sigma and trying to figure out some
way to constrain it to be positive – even though pytorch doesn’t naturally
do constrained optimizations – you train the possibly-negative (and
unconstrained) q_parameter, but then map it to an always-positive

Good luck.

K. Frank

1 Like

Thank you, it does indeed make sens