Different input for Multitask Model? How to do it?

1

I have read a paper about a fancy multitask model, and I am trying to rebuild it through pytorch. While I have encountered with several questions. In the model(shown below), the multitask model has two part of input, and they share an encoder. As a freshman, I am a little confused about how to build the model.

I have some thoughts, but I don’t whether they would work.
Should I input all the inputs into one model but separate them into different parts?
But if I do this, how can i solve the problem that the two datasets have different size?

Thank you very much!!!

the model:

image
image1270×896 65.3 KB

2

the encoder in the model is the same

3

Assuming the encoder is the same one, simply compute forward outputs. If the size of inputs from different datasets is different, conventionally, you need to make them the same size such as cropping them or resizing them to the same size.

I write a brief example for you reference.

enc = model()
fc = nn.Linear(...)
DNN = another_model()

input1 = torch.ones(batch_size, 10)
input2 = torch.zeros(batch_size, 10)

optim = optim.Adam(enc.parameters(), lr=lr)

... some operation for input1
output1 = fc(enc(input1))

... some operation for input2
output2 = DNN(enc(intput2))

loss_1 = cal_loss1(output1, target1)
loss_2 = cal_loss2(output2, target2)

total_loss = loss_1 + loss_2

optim.zero_grad()
total_loss.backward()
optim.step()
4

Thank you very much!!! Your suggestion is such a help for me !