As I understood, the Tutorial for Parameter server based on the RPC framework is a special implementation based on different assumptions.
1- The data should be sent to the parameter server (for a large dataset is it possible???)
2- it Async
3- distributed auto grad has been used
am I correct?
I am wondering is there any way to implement a sync parameter server with the RPC framework without moving data to the parameter server? I mean just use local grad and for example, model averaging on parameter server. any exist?