# Mutual / Lateral inhibition in a single layer

Hi,

Is there a away to apply mutual / lateral inhibition, in a linear layer, where there is only one winner (value near 1) and the rest are inhibited (near 0) in a differential-able way ?

Thanks

Hi Karim!

If you want to have `n` mutually-inhibitory values, `x_i`, you can

The idea is that you want your loss to be small (say, zero) when
you are at (or near) the `n` special points you prefer. (These `n`
points are where one of the `n` `x_i` is 1, and the others are zero.)

So we take the (squared) distance of your actual `x_i` from each
of those preferred points and multiply those `n` distances together.
Because you multiply them all together, being close to any of the
preferred points makes your loss small, and being exactly at one
of those points makes your loss exactly zero.

Here’s a formula for the above words:

``````inh = Prod_{i} ( (x_i - 1)^2 + Sum_{j != i} (x_j)_^2 )
``````

We built this inhibition term, `inh`, out of differentiable pieces, and
combined them together in a differentiable way, so `inh` is fully
differentiable.

(I assume that you want the output of your linear layer to display this
inhibition, rather than, say, the parameters, but this scheme doesn’t
depend on where the `x_i` come from.)

Good luck.

K. Frank

1 Like