How to handle two separate optimizers and separate losses?

[JimW]

Sorry for my terrible drawing. But according to the above figure, I want to use 2 separate losses to update different modules in the entire network. The data flow is illustrated with arrows in the above figure, I hope to define a loss1 to only update “trainable 1”, and another term loss2 to only update “trainable 2”.

I apologize because I can’t paste our internal code here, so I have to briefly mention what I did:
I defined two optimizers: optimizer_1 (with parameters in “trainable 1” as trainable variables) and optimizer_2 (with parameters in “trainable 2” as trainable variables).

For each iteration of my training loop, I did something like this:

optimizer_1.step()
optimizer_2.step()
loss1.backward()
loss2.backward()

And crashed with the following error message
# [RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time]

I am not sure if I should handle these two losses in this way. Could someone tell me what might be potentially wrong in my implementation, and how to fix it?

Thanks.

[Programmer-RD-AI]

Two optimizers for one model

Is there a way to use two optimizers for one model in more beautiful way? Now, as I understand we should do something like these: net = Model() part_1_parameters = ... part_2_parameters = ... opt1 = optimizer_1(part_1_parameters) opt2 = optimizer_2(part_1_parameters) ### train epoch opt1.zero_grad() opt2.zero_grad() loss = ... loss.backward() opt1.step() opt2.step()

How to run with multiple GPU when having 2 optimizers?

I think I totally understood the tutorial on using the following line of code to do data parallel. model = nn.DataParallel(model) I tested it on my device with 2 GPUs and it worked. However, for a complicated model I recently wrote, it doesn’t truly train on both GPUs when I printed the ‘Inside’ ‘Outside’ debugging message. I managed to figure out that it was because of the optimizers. To my purpose, I need to have two optimizers each contains some shared part and some unique part. model = …

Correct way to declare multiple optimizers for single model in DDP

Given the pseudo model below: class model(nn.Module): def __init__(self): self.alpha_params1 = nn.Parameters(<size>, requires_grad=True) self.alpha_params2 = nn.Parameters(<size>, requires_grad=True) < typical Conv2d layers>.... def forward(self, x): <feed forward> return output def net_parameters(self, filtered_name='alpha_', recurse=True): # return torch layer params for name, param in self…

check the above discussions

[JimW]

Thank you so much for the pointers.
Could you also comment on this “# [RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time]” error?

I tried to use splited parameters (i.e. trainable_1 assigned to optimizer1, and trainable_2 assigned to optimizer2, while there should not be overlap btw trainable_1 and trainable_2) for different optimizers to separate the two training flow. Where could be wrong to trigger this “Trying to backward through the graph a second time” error? BTW, I don’t fully understand this error, and will “retain_graph=True” be a correct fix in this case?

[my3bikaht (Sergey)]

When you do backward pass, some of the values of the graph for the intermediate (non-leaf) nodes are freed. So, second pass over same nodes will trigger this error. To be able to proceed with second pass, you need to add parameter to loss1.backward(retain_graph=True)