Two networks with one loss. How is it possible?

1

Hi all, I have two distinct network with same loss and two optimizers for each network. How can I backprop the loss to the networks?
Any help would be appreciated.

        optimizer_A.zero_grad()
        optimizer_B.zero_grad()
        loss.backward()
        optimizer_A.step()
        optimizer_B.step()
2

Hi,
This probably means the common loss is a function of the parameters of both the models.

You could then just create a single optimizer containing the parameters of both the models, like so:

optimizer = torch.optim.Adam([modelA.parameters()] + [modelB.parameteres()])

Let me know if this gives any errors.

3

Hi @srishti-git1110, thanks for your tip, but my optimizers are two distinct optimizers with different hyperparameters.

4

In that case, is this code not working?

There’s also an option to create parameter groups within the same optimizer but I think that is equivalent to creating two different optimizers with different learning rates etc. (even for optimization algorithms like Adam that calculate running stats.)

6

Is it true? I mean what is the acceptable code for this problem

7

Your code should work fine. Please post if you face any errors.

1 Like