Combine network parameters from different networks?

1

How does one combine network parameters from two different networks?

Suppose I have two (could be more but let’s do two) distinct networks model1 and model2. I would like the optimizer to be aware of and optimize both models simultaneously.

I want to do something like this:

import torch.optim as optim
optimizer = optim.SGD(model1.parameters() + model2.parameters(), lr=0.001, momentum=0.9)

An example use case is where model1 is pretrained and potentially also model2.

Any ideas or suggestions?

2

This might work for anyone who is wondering.

import itertools
optimizer = optim.SGD(itertools.chain(model1.parameters(), model2.parameters()), lr=0.001, momentum=0.9)
4 Likes
3

another (actually worse) method:

optimizer = optim.SGD([p for p in model1.parameters()] +
                      [p for p in model2.parameters()]),
                      lr=0.001, momentum=0.9)