Hi i want to seed my env
when i use
model = model.cuda()
my model seeded correctly and i can reporduce the results
but when i try to use :
model = nn.DataParallel(model,device_ids=[0,1]).cuda()
i have a small variability in my results
Hi i want to seed my env
when i use
model = model.cuda()
my model seeded correctly and i can reporduce the results
but when i try to use :
model = nn.DataParallel(model,device_ids=[0,1]).cuda()
i have a small variability in my results
Did you follow all instructions mentioned in the Reproducibility Docs?
ofc i use this to seed model init and other numpy related funcs
random_seed = 42
random.seed(random_seed)
torch.manual_seed(random_seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(random_seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
for data loading processes seed i use this
import numpy
import random
def seed_worker(worker_id):
worker_seed = torch.initial_seed() % 2**32
numpy.random.seed(worker_seed)
random.seed(worker_seed)
seed = 42
generator = torch.Generator()
generator.manual_seed(seed)
......
self.train_loader = torch.utils.data.DataLoader(
trainset,
sampler=build_train_sampler(
trainset.train,
train_sampler,
batch_size=batch_size_train,
num_instances=num_instances,
),
batch_size=batch_size_train,
shuffle=False,
num_workers=workers,
drop_last=True,
generator=generator,
worker_init_fn=seed_worker,
)
i think the problem is related to parallel preoceeses lunched useing nn.DataParallel
I might be missing it, but I don’t see e.g. torch.use_deterministic_algorithms(True)
as mentioned in the docs.
Why do you think that’s the case?