How does pytorch ensure numerical stability for sigmoid? Python can handle upto approx 1.7e+709. How is this handled? I tried to go through the source code but couldn’t find the implementation for sigmoid.
You can change the torch.dtype to torch.float64 for all operations via torch.set_default_dtype(torch.float64) (docs: torch.set_default_dtype — PyTorch 2.0 documentation)
If you want to see the source code for torch.Sigmoid, you’ll have to look for it on the GitHub repo (I’m sure someone will know the path to the file).
As a side note, you can always represent your problem within the log-domain and that’ll have a much better range than doing it within the linear-domain.