Why softmax training is more stable (than sigmoid)

[laro (amit)]

I’m wondering about which activation function will be easier to train with (get better accuracy / smallest loss) - with SoftMax or sigmoid (for multiclass classification problem)

According to: https://www.quora.com/What-are-the-benefits-of-using-a-softmax-function-instead-of-a-sigmoid-function-in-training-deep-neural-networks

Training model for multiclass with SoftMax - the training is more stable vs training with sigmoid

• Why is it true ?
• Is it easier to train model (and get better results) with SoftMax (instead of sigmoid) ?

[KFrank]

Hi Amit!

@laro

I’m wondering about which activation function will be easier to train with (get better accuracy / smallest loss) - with SoftMax or sigmoid (for multiclass classification problem)

My reply to your other post probably answers this question. See:

Which activation function for multi-class classification gives true probability?

Hi Amit! The short answer is that softmax() and sigmoid() are used for different things. It’s not that one is true and the other false or that one is more stable and the other less stable – they’re just different. Let me give you my perspective on this. (I haven’t looked at the links you posted.) For some context, my number-one rule is that the output of a neural network means whatever you train it to mean. Let’s say that you have a problem with three classes and the final layer of your…

Best.

K. Frank