How to get deterministic behavior?

22

I’m not sure why… but this does not work for me…

23

I never got this to work, no. I’ve been doing this on Windows, though. Not sure if that makes a difference. Some PyTorch functionality isn’t optimized for Windows I’ve found.

24

Hi, I have used exactly the same code but still my code non-deterministic. I am not able to reproduce the results. Tried in both pytorch 1.4.0 and 1.3.1

Thanks.

25

Hello, In my case It was about using the Parallel from joblib to cache the dataset. Replacing Parallel with for loop helped

26

Hello, I tried all the options suggested above but still am unable to get deterministic behavior. Here is a part of my code:

seed = 1
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

# When running on the CuDNN backend, two further options must be set
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

#For tensorflow
os.environ['TF_CUDNN_DETERMINISTIC'] = '1'
os.environ['TF_DETERMINISTIC_OPS'] = '1'
	
# Set a fixed value for the hash seed
os.environ["PYTHONHASHSEED"] = str(seed)

def worker_init_fn(worker_id):
    numpy.random.seed(seed)
    tf.random.set_seed(seed)
    tf.compat.v1.set_random_seed(seed)
    random.seed(seed)

g = torch.Generator()
g.manual_seed(seed)

And my dataloader is:

content_iter = iter(data.DataLoader(
    content_dataset, batch_size=args.batch_size,
    sampler=InfiniteSamplerWrapper(content_dataset),
    num_workers=0, shuffle=False,
    worker_init_fn=worker_init_fn,
    generator=g))

style_iter = iter(data.DataLoader(
    style_dataset, batch_size=number_of_styles,
    sampler=InfiniteSamplerWrapper(style_dataset),
    num_workers=0, shuffle=False,
    worker_init_fn=worker_init_fn,
    generator=g))

The InfiniteSamplerWrapper() function is defined as:

import numpy as np
from torch.utils import data

def InfiniteSampler(n):
    i = n - 1
    order = np.random.permutation(n)
    while True:
        yield order[i]
        i += 1
        if i >= n:
            order = np.random.permutation(n)
            i = 0

class InfiniteSamplerWrapper(data.sampler.Sampler):
    def __init__(self, data_source):
        self.num_samples = len(data_source)

    def __iter__(self):
        return iter(InfiniteSampler(self.num_samples))

    def __len__(self):
        return 2 ** 31

Have I missed setting the seed for anything?

27

It seems you are not setting the numpy seed besides in the worker_init_fn, which won’t be used since you are setting num_workers=0 in your DataLoaders.

1 Like
28

Thanks much. It’s working now.