Torch.sum does not benefit from parallelism?

[anantguptadbl (Anant Gupta)]

@Louka_Ewington-Pitso

for bs in [4, 8, 16, 32, 64, 128, 256]:
    tensor = torch.randn(bs, 512, 25000).to('cuda')

    start = time.time()
    for i in range(1000):
        torch.sum(tensor)

    print(time.time() - start)

@ptrblck Can you share some links etc that would explain the CUDA parallelization that you mentioned