Softmax don't give sum to 1 when batch size is big

import torch
import torch.nn as nn
batch_size = 16
a = torch.rand(batch_size,12,4864,4864).cuda()
m = nn.Softmax(dim=3)

when batch_size is big, the output is not 1, when I change batch_size to 8, The output is all 1.
Is there any way to make the calculation of the softmax function more accurate?

Hi Xingsen!

How inaccurate is Softmax in your test? You are summing over a
large number of values, and it is to be expected that some floating-point
round-off error occurs.

Look at the largest deviations from the expected value of 1.0. Do your
results agree with 1.0 up to about 6 or 7 decimal places – that is, within
expected floating-point round-off error?

You can make the calculation of the softmax function more accurate by
performing the calculations in double-precision.


K. Frank