You can use the share_memory() function on an nn.Module so that the same parameters can be accessed from multiple processes (using the multiprocessing module). If I do not want these parameters to be shared, and I want each subprocess to get an independent copy of the parameters to work with, will simply not calling share_memory() provide this behavior?
4 Likes
no, what you need to do is to send them model to the new process, and then do:
import copy
model = copy.deepcopy(model)
share_memory() only shares the memory ahead of time (in case you want to reuse the shared memory for example)
7 Likes
Or, assuming your model is a module called MyModel, you can create a separate instance in each process, send a state_dict from a single one to everyone else, and have them load_state_dict. The parameters in the state_dict will be shared among the processes, but load_state_dict only copies the content, without reassigning tensors, so your model won’t be shared.
However, if you want to duplicate models within a single process, I definitely recommend the way proposed by @smth.
4 Likes
Then could you give me an example of using load_state_dict() to copy a module?
He is referring to saving the state_dict (for instance into a file), sending it over to another process, then loading it with load_state_dict