I have some numerical issues and I am trying to find out if there is a bug in my calculation or not. I am using a model which samples weights and therefore I am using logsumexp over the sample dimension such as this if we assume I have a tensor of
ll = torch.logsumexp(log_lik.sum(dim=1), dim=0) ll = ll - np.log(sample_n) ll = ll / instances
My predictions are extremely low probability because they are out of distribution samples so the log probability value can be something like
-100,000. When I try to do the logsumexp operation without using the torch function then
exp(-100000) just gives 0.
In this case I would expect that the output of logsumexp would then be
-inf since it is taking the log of
0, but it always comes up with some answer. How can this be?