# Adam optimizer with softmax gives grad\_fn=\<SoftmaxBackward0\>

**URL:** <https://discuss.pytorch.org/t/adam-optimizer-with-softmax-gives-grad-fn-softmaxbackward0/204083>\
**Category:** Uncategorized\
**Created:** [June 5, 2024, 10:22am UTC](https://discuss.pytorch.org/t/adam-optimizer-with-softmax-gives-grad-fn-softmaxbackward0/204083 "2024-06-05T10:22:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![undefined](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/undefined/32/70420_2.png) [@undefined](https://discuss.pytorch.org/u/undefined)\
**Post date:** [June 5, 2024, 10:22am UTC](https://discuss.pytorch.org/t/adam-optimizer-with-softmax-gives-grad-fn-softmaxbackward0/204083/1 "2024-06-05T10:22:40Z")

</div>

I’m using the Adam optimizer for backpropagation. In my last layer of my network I have the SoftMaxfunction. I’m using CrossEntropyLoss. I know that SoftMaxwith CrossEntropyLoss is not the best choise as CrossEntropyLoss already performs SoftMax, but in this case the model output gives values between 0 and 1.  
When feeding the network one sample, I get an output like this:

```auto
Model output: tensor([[1.0000e+00, 1.4013e-45]], device='cuda:0', grad_fn=<SoftmaxBackward0>)

```

But when I remove SoftMax from the last layer, I get:

```auto
Model output: tensor([[-5.9609, 4.6506]], device='cuda:0', grad_fn=<AddmmBackward0>)

```

What I have seen is that without SoftMax training works faster and loss decreases to almost 0.

I would like to know if there is a difference between these two cases? Why is grad\_fn different when I’m using Adam optimizer in both cases?

---

<div class="post-metadata">

**Author:** ![AlphaBetaGamma96](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/32.png) [@AlphaBetaGamma96](https://discuss.pytorch.org/u/AlphaBetaGamma96)\
**Post date:** [June 5, 2024, 12:11pm UTC](https://discuss.pytorch.org/t/adam-optimizer-with-softmax-gives-grad-fn-softmaxbackward0/204083/2 "2024-06-05T12:11:00Z")

</div>

Hi @undefined,

> [@undefined](#):
>
> I would like to know if there is a difference between these two cases?

When using the SoftMax function, you’re predicting a class given an input (which for your model predicts the first class, out of 2 available classes).

> [@undefined](#):
>
> ```auto
> Model output: tensor([[1.0000e+00, 1.4013e-45]], device='cuda:0', grad_fn=<SoftmaxBackward0>)
> 
> ```

In the second case, you’re just outputting the logits and if you fit that to your model, you’re no longer predicting classes based on a probability, but just fitting the output (which isn’t the same).

If you want to read more info, there’s a nice thread with more information here: [Logits vs. log-softmax - #2 by KFrank](https://discuss.pytorch.org/t/logits-vs-log-softmax/95979/2)

---

<div class="post-metadata">

**Author:** ![undefined](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/undefined/32/70420_2.png) [@undefined](https://discuss.pytorch.org/u/undefined)\
**Post date:** [June 5, 2024, 3:32pm UTC](https://discuss.pytorch.org/t/adam-optimizer-with-softmax-gives-grad-fn-softmaxbackward0/204083/3 "2024-06-05T15:32:52Z")

</div>

As I understood it correct using SoftMax with CrossEntropyLoss is bad practice. Thats why I wont use SoftMax. Another thing I’m curious about is which loss function to use as I have 2 classes? CrossEntropyLoss or BCEWithLogitsLoss?