Gradient scaling in federated learning

Hi, I am using torch.distributed to do federated learning. What should I do if I want to scale some workers’ gradient since their data are more valuable than others? Thx a lot in advance!

tensor.register_hook(customHook) may work, you need to write customHook to modify grad of the tensor.

but as far as I know customHook should be a function of grad only. For your case, you want to make customHook to be a function of grad and workerRank as well?

I only want to make customHook to be a function of grad and I haven’t used this before. Does hook only influence the backward process?

yes, only for backward pass