Multiprocess Sample all identical - No Randomness

When running a model and then sampling from the output distribution in multiple process with the multiprocessing module, every sample is the same even though they should be random. What is the reason for the sample being identical? A simple working example is:

import multiprocessing
import torch
import torch.nn as nn

class test(multiprocessing.Process):
    def __init__(self, weights):
        multiprocessing.Process.__init__(self)
        self.m = nn.Linear(10, 10)
        self.m.load_state_dict(weights)
        
    def run(self):
        out = self.m(torch.ones(10))
        dist = Categorical(logits=out)
        print(dist.sample())

def main():
    orig_model = nn.Linear(10,10)
    weights = orig_model.state_dict()
    workers = [test(weights) for i in range(100)]
    for w in workers:
        w.start()

    for w in workers:
        w.join()

with output

tensor(2)
tensor(2)
...
tensor(2)
tensor(2)

Looks like you are override the weight from a global weight before spawning multiple processes. Or am I missing something here?

Yes, I override the weights so each worker has identical weights. But that shouldn’t affect the randomness. It just means every worker has the same distributions - not that every sample from the same distribution should be the same

Ah I see. Does use torch.manual_seed(self.rank) inside your run help here?

Inside the run function it helps. But inside __init__ it does not help

1 Like