KFrank
(K. Frank)
April 4, 2020, 4:55pm
2
Hi Ajinkya!
You want a differentiable, “soft” threshold function. I think that
nn.Tanhshrink gives you most of what you want. You can put in
a threshold parameter like this:
threshold * nn.functional.tanhshrink (x / threshold)
I’ve never done this, but I believe that if you make your
threshold
value a nn.Parameter
and include it in your model,
it should work.
The method in this post:
You could write your own nn.Module that includes a trainable alpha parameter:
class MyLayer(nn.Module):
def __init__(self, *args, **kwargs):
self.alpha = nn.Parameter(torch.tensor(0.))
# Other necessary setup
super().__init__(*args, **kwargs)
def forward(self, x):
# Necessary forward computations
return output
Using this MyLayer module as part of a network architecture will allow you to learn alpha along with all of the other usual parameters. …
looks plausible to me (but I can’t vouch for it).
Best.
K. Frank