Computationally stable log-softplus

[KFrank]

Hi Steve!

@stevethesteve

This changes things a lot: If setting the breakpoint too low, the error in the gradient can be 7 orders of magnitude bigger than the error in the log-softplus output.

This may be related to some weirdness in pytorch’s built-in
torch.nn.functional.softplus(). Here is the relevant thread:

Gradients for torch.nn.functional.softplus() are numerically unstable

Hello Forum! It appears that backward() for softplus() has a numerical issue (that does not appear to be caused directly by a related forward() issue) for large negative arguments. A straightforward (but not completely naive) implementation of softplus() – that has no explicit backward() – seems to work fine with autograd. Here is an illustrative test script: import torch print (torch.__version__) def test_softplus (x): # use log-sum-exp trick and loq1p return torch.where (x > 0,…

Best.

K. Frank