What's the meaning of gain in initial

ninesun127 · February 19, 2019, 1:28pm

![image|423x500]

what is the meaning of the gain?
what‘s the differences between gain=1 and gain=sqrt(2)

albanD · February 19, 2019, 1:30pm

Hi,

This is explained in the doc for init.calculate_gain.

ninesun127 · February 19, 2019, 1:50pm

thank you.i have saw the doc, however i really do not know how it works

albanD · February 19, 2019, 2:16pm

Hi,

From what I remember, It is just a heuristic that depends on the non linearity.
It represent roughly how the non linearity scale the outputs and by using this information, you can initialize your weights in a better way. To make sure your network will keep everything at roughtly the same scale.