in the above image, the top cell is showing my softmax predictions, but when i index into it (to get the preds for the first input) i am not getting the exact same numbers (as shown in the cell below).
i want to use integer array indexing for calculating the neg. log likelihood but since the preds are not same, the NLL is coming ‘inf’.
grad_fn changes since indexing is a differentiable operation.
You are getting the same values, but the printing uses another format (e.g.
-98.1084 instead of
-9.8108e+01) since the range of the slice is smaller and wouldn’t benefit from the “scientific” format.
That’s not true and the
-inf value is also in the original
sm_pred tensor but just not displayed.
i am re-implementing a notebook from 2019…this is the
function and the
function… i am just using the exact same code from the original notebook but still my
are not the same… has any functionality used in these functions changed during these years???
I’m sure a lot of things were moved around between 2019 and now, but I don’t know the exact changes in your used operations.
However, your current manual log_softmax implementation is numerically not stable and I would recommend to use
F.log_softmax which might be able to avoid the invalid outputs assuming the input does not contain any invalid values (it did before).
Yeah i understand. But i am implementing it from scratch just to deepen my understanding of the pytorch library. At the end, i use the
F.log_softmax function for all my projects.
I understand the idea, but in this case it would be your responsibility to check the numerical stability as your current implementation can create invalid outputs more easily:
return (x.exp() / (x.exp().sum(-1,keepdim=True))).log()
x = torch.tensor([[-100., 100.]])
out = log_softmax(x)
# tensor([[-inf, nan]])
out = F.log_softmax(x, dim=1)
# tensor([[-200., 0.]])