I can't seed with nn.DataParallel

1

Hi i want to seed my env
when i use

model = model.cuda()

my model seeded correctly and i can reporduce the results
but when i try to use :

model = nn.DataParallel(model,device_ids=[0,1]).cuda()

i have a small variability in my results

2

Did you follow all instructions mentioned in the Reproducibility Docs?

3

ofc i use this to seed model init and other numpy related funcs

random_seed = 42
random.seed(random_seed)
torch.manual_seed(random_seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(random_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

for data loading processes seed i use this

import numpy 
import random
def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

seed = 42
generator = torch.Generator()
generator.manual_seed(seed)

...... 

self.train_loader = torch.utils.data.DataLoader(
    trainset,
    sampler=build_train_sampler(
        trainset.train,
        train_sampler,
        batch_size=batch_size_train,
        num_instances=num_instances,
    ),
    batch_size=batch_size_train,
    shuffle=False,
    num_workers=workers,
    drop_last=True,
    generator=generator,
    worker_init_fn=seed_worker,
    
)

i think the problem is related to parallel preoceeses lunched useing nn.DataParallel

4

I might be missing it, but I don’t see e.g. torch.use_deterministic_algorithms(True) as mentioned in the docs.

Why do you think that’s the case?