Hi,
Is there a away to apply mutual / lateral inhibition, in a linear layer, where there is only one winner (value near 1) and the rest are inhibited (near 0) in a differential-able way ?
Thanks
Hi Karim!
If you want to have n mutually-inhibitory values, x_i, you can
add an “inhibition” loss term to your overall loss function.
The idea is that you want your loss to be small (say, zero) when
you are at (or near) the n special points you prefer. (These n
points are where one of the n x_i is 1, and the others are zero.)
So we take the (squared) distance of your actual x_i from each
of those preferred points and multiply those n distances together.
Because you multiply them all together, being close to any of the
preferred points makes your loss small, and being exactly at one
of those points makes your loss exactly zero.
Here’s a formula for the above words:
inh = Prod_{i} ( (x_i - 1)^2 + Sum_{j != i} (x_j)_^2 )
We built this inhibition term, inh, out of differentiable pieces, and
combined them together in a differentiable way, so inh is fully
differentiable.
(I assume that you want the output of your linear layer to display this
inhibition, rather than, say, the parameters, but this scheme doesn’t
depend on where the x_i come from.)
Good luck.
K. Frank
1 Like