Initialization gain for SiLU activation

[Lelouch1 (Lelouch)]

As the current function torch.nn.init — PyTorch 1.13 documentation does not give the gain for the SiLU activation SiLU — PyTorch 1.13 documentation , do you know what value I should use ?

[tom (Thomas V)]

I guess it boils down to how much and what kind of work you want to put into it.
Some ideas:

• “if it is close to ReLU” use ReLU gain,
• check the literatur (links given in the SiLU documentation),
• experiment yourself.

For the last one, maybe the discussion we had regarding SELU might be some inspiration:

Calculate_gain('tanh')

So I know nothing about these the paper you cite. And I haven’t looked at the papers for He’s and Glorot’s initialization, but here is Thomas Theory of Weight Initialization for the Theory Adverse™: Why do we care about initialization? We want “stability” of activation distributions in deep networks. If no “signal” arrives in layer 100, we are screwed. How do we measure “signal”? Let’s just take standard deviation. What is then a good gain? One where the standard deviation “converges” to a rea…

Of course, we (I) got scolded by the people who prefer computing the gain from the fixed point calculation, but if your goal is to train networks successfully, that might not be as crucial.

Best regards

Thomas