How to handle two separate optimizers and separate losses?

[Programmer-RD-AI]

Two optimizers for one model

Is there a way to use two optimizers for one model in more beautiful way? Now, as I understand we should do something like these: net = Model() part_1_parameters = ... part_2_parameters = ... opt1 = optimizer_1(part_1_parameters) opt2 = optimizer_2(part_1_parameters) ### train epoch opt1.zero_grad() opt2.zero_grad() loss = ... loss.backward() opt1.step() opt2.step()

How to run with multiple GPU when having 2 optimizers?

I think I totally understood the tutorial on using the following line of code to do data parallel. model = nn.DataParallel(model) I tested it on my device with 2 GPUs and it worked. However, for a complicated model I recently wrote, it doesn’t truly train on both GPUs when I printed the ‘Inside’ ‘Outside’ debugging message. I managed to figure out that it was because of the optimizers. To my purpose, I need to have two optimizers each contains some shared part and some unique part. model = …

Correct way to declare multiple optimizers for single model in DDP

Given the pseudo model below: class model(nn.Module): def __init__(self): self.alpha_params1 = nn.Parameters(<size>, requires_grad=True) self.alpha_params2 = nn.Parameters(<size>, requires_grad=True) < typical Conv2d layers>.... def forward(self, x): <feed forward> return output def net_parameters(self, filtered_name='alpha_', recurse=True): # return torch layer params for name, param in self…

check the above discussions