Set seed doesn't work

vision
1

Hello everyone. I need to create a simple CNN and incrementally add pieces for a classification problem. Even though I’ve tried setting the seed in every operation involving randomness, rerunning the same cell still gives me different results.

Below is what I did to limit randomness:

def set_seed(seed=0):
    """Set seeds for reproducibility"""
    # Python random
    random.seed(seed)
    
    # NumPy
    np.random.seed(seed)
    
    # PyTorch
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)  # If using multi-GPU

    # CuDNN 
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    
    # Python hash
    import os
    os.environ['PYTHONHASHSEED'] = str(seed)
    
set_seed(42)  # Set the seed for reproducibility

Here for data loading:

# Set up a generator for reproducibility
g = torch.Generator()
g.manual_seed(42)

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32

# 3. Split train and validation sets
train_subset, val_subset = random_split(train_dataset, [train_size, val_size], generator=g)

# 4. Data Loaders
train_loader = DataLoader(train_subset, batch_size=32, shuffle=True, worker_init_fn=seed_worker, generator=g)
val_loader = DataLoader(val_subset, batch_size=32, shuffle=False, worker_init_fn=seed_worker, generator=g)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False, worker_init_fn=seed_worker, generator=g)

And here is also the initialization of the weights:

def my_init_weights(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        # Gain weights from a gaussian distribution
        nn.init.normal_(m.weight, mean=0.0, std=0.01)
        # Bias at 0
        if m.bias is not None:
            nn.init.zeros_(m.bias)

# Apply to the model
model.apply(my_init_weights)

I know I’ve been a little overkill, but it’s become a matter of principle. Is there still something random that affects my results? Any other tips is appreciated

2

Maybe you could check that the initial weights are always the same (e.g. save those weights right after initialization before training, then rerun, save the new weights, and compare them). Then you could check if the loaded data is always the same, and lastly if its not the initialization nor the data loading, you could try to pinpoint which step of the forward or the backward pass is non-deterministic. This way, at least you’ll know what operation is the source of your problem.

A few ideas of the possible causes:
It’s possible that something is wrong in your code (e.g. the generator you create is for cpu device, but maybe you use that generator for a random operation happening on cuda), that some pytorch operation is non-deterministic (even though it’s rare in my experience, and this would deserve opening an issue), or that the hardware you run your code onto is not always the same at each run, due to automatic cpu or gpu assignment that would differ between runs.

1 Like
3

I don’t see the usage of torch.use_deterministic_algorithms() as described in the Reproducibility docs so you might need to add it.

1 Like
4

Thank you for your help.

At the end none of these were the actual solution but you helped me understanding what was the problem. Since I’m working on a Jupyter Notebook each cell need to reset the seed in order to achive reproducibility. I’m still trying to understand why, since every seed should be globally set, but for now the problem is solved.