Is it normal that pytorch is not 100% deterministic even when seeding all of my sources of randomness?

1

I was seeding my code with:

    seed = args.seed
    if seed is None: # if seed is None it has not been set, so get a random seed, else use the seed that was set
        seed = ord(os.urandom(1))
    print(f'seed: {seed}')
    torch.manual_seed(seed)

then I trained my models but the final errors aren’t exactly the same (they are the same in about 7 significant figures):

one run:

[1, 196], (train_loss: 2.080742538583522, train error: 0.7419323979591836) , (test loss: 1.9844910830259324, test error: 0.70859375)

another:

[1, 196], (train_loss: 2.0807455449688192, train error: 0.7419523278061224) , (test loss: 1.9844940572977066, test error: 0.70859375)
2

You still have cudnn random algorithms (if using cudnn) and data loader worker multiprocessing randomness (if using num_workers > 0).

For the former, you can set torch.backends.cudnn.deterministic=True. For the later, set num_workers=0 or build from a commit after https://github.com/pytorch/pytorch/pull/4640.

1 Like
3

how much speed am I trading off for doing that? num=0 and determinism in cudnn?

4

It depends on your task. Maybe you should try it out. It shouldn’t be hard to measure.