How to deal with inf values when taking logs of tensors containing zeros

1

In my model, in the forward function, I have one tensor on which I first apply sigmoid and then log. However, the output of sigmoid might be considerably low. The return value of the forward function is :

def forward(dots, path, probs)
        apply_paths = dots * path 
        apply_sigmoid = torch.sigmoid(apply_paths)
        apply_log = -1*torch.log(apply_sigmoid)

       return torch.sum(probs*apply_log)

I’m getting a lot of time inf as a return value. The return value is the loss itself, and so the gradients are calculated based on this return value. I’m not sure if this is okay or not. Should I do anything of these inf values after taking log? Or do something about very small values from the sigmoid function?

2

You could clamp these invalid values or just use logsigmoid as seen here:

x = torch.tensor([-100., 100])
print(torch.log(torch.sigmoid(x)))
> tensor([-inf, 0.])
print(torch.nn.functional.logsigmoid(x))
> tensor([-100.,   -0.])
1 Like