Hi,

Is there a away to apply mutual / lateral inhibition, in a linear layer, where there is only one winner (value near 1) and the rest are inhibited (near 0) in a differential-able way ?

Thanks

Hi,

Is there a away to apply mutual / lateral inhibition, in a linear layer, where there is only one winner (value near 1) and the rest are inhibited (near 0) in a differential-able way ?

Thanks

Hi Karim!

If you want to have `n`

mutually-inhibitory values, `x_i`

, you can

add an â€śinhibitionâ€ť loss term to your overall loss function.

The idea is that you want your loss to be small (say, zero) when

you are at (or near) the `n`

special points you prefer. (These `n`

points are where one of the `n`

`x_i`

is 1, and the others are zero.)

So we take the (squared) distance of your actual `x_i`

from each

of those preferred points and multiply those `n`

distances together.

Because you multiply them all together, being close to any of the

preferred points makes your loss small, and being exactly at one

of those points makes your loss exactly zero.

Hereâ€™s a formula for the above words:

```
inh = Prod_{i} ( (x_i - 1)^2 + Sum_{j != i} (x_j)_^2 )
```

We built this inhibition term, `inh`

, out of differentiable pieces, and

combined them together in a differentiable way, so `inh`

is fully

differentiable.

(I assume that you want the output of your linear layer to display this

inhibition, rather than, say, the parameters, but this scheme doesnâ€™t

depend on where the `x_i`

come from.)

Good luck.

K. Frank

1 Like