Hi, I am using torch.distributed to do federated learning. What should I do if I want to scale some workers’ gradient since their data are more valuable than others? Thx a lot in advance!
tensor.register_hook(customHook) may work, you need to write customHook to modify grad of the tensor.
but as far as I know customHook should be a function of grad only. For your case, you want to make customHook to be a function of grad and workerRank as well?
I only want to make customHook to be a function of grad and I haven’t used this before. Does hook only influence the backward process?
yes, only for backward pass