I am trying to implement federated learning for my research. However, the tutorials or code snippets that I found (on MNIST) distribute a collected MNIST dataset over n number of clients.
To make this simple, suppose I have just 2 clients with different data as inputs(for eg. client A has data 1-5 and client B has data 6-10). I want to train the model separately on Client A and Client B with their respective data (not distributing the combined dataset). Is this possible to do so? can you please give me an example to proceed further?
To clarify on my current setup, I am using Windows 10 with pytorch 1.0.0 and do not know if additional libraries like pysyft works for my version. I just want to make it work using pytorch.
I assume Client A and Client B need to communicate (maybe through a master) at some point to average the model? What communication pattern (e.g., collective vs RPC) will they be using?
We do have collective and RPC API, but they are not yet available on Windows. If you need this on Windows, could you please comment on this poll to help us better prioritize it?
Yes it’s just like in federated learning. client A and client B will be training their own data for respective number of epochs E. Then they need to send the gradients to a master who will average the model and send it back to the clients. I do have some access to Linux system. I would appreciate if you could provide me an example to do this.
P.S. Can you tell how to make sure that each of the model has different input? currently all of the examples use a same dataloader to get the input samples and just randomly distribute the data as input. I need to have each of the model train separately on their own data.
Since this is master/client training scenario, it might fit into the parameter server paradigm. Please see this tutorial and this example for a complete example.
Can you tell how to make sure that each of the model has different input?
In this case, as each client is running in its own process, it can create its dedicated dataloader. Or if you need the master to dispatch the input data to those clients, this is also doable. You can wrap those client logic into a function and let the master to launch them, sth like rpc.rpc_async("client_a", client_func, args=(inputs_for_a,)).
I need to have each of the model train separately on their own data.
Since the forward-backward does not need to span across machine boundaries, this RPC+RRef Example might be sufficient, as you don’t need distributed autograd and distributed optimizer.