I have a model that uses two preexisting models to extract representations. These representations are passed into a sequential model and the whole thing is trained end-to-end. Or should be, but the weights in the two representation extraction models aren’t updating. A simple example might look something like this:
class Net(nn.Module):
def __init__(self, model1, model2):
super(Net, self).__init__()
self.model1 = model1
self.model2 = model2
self.classifier = nn.Sequential(nn.Linear(h1, h1), nn.ReLU(), nn.Linear(h1, h2))
def forward(self, x):
x1 = self.model1(x)
x2 = self.model2(x)
out = self.classifier(torch.cat([x1, x2], dim=1))
return out
Am I doing something obviously wrong here?