Can I collectivize models learning?


I saw some topics like this one

My case is little more complex, I simplify it:

I have data, some of them have common parts but not all. There is not common part for all data

For example,

I have data of type A with features F0 … F6
data of type B with features F0…F3 and F7…F8
data of type C with features F6…F9

An illustration:

We can see all data have the same size. My idea is to do little neural networks for all subsets of features (S0 = F0…F3, S1 = F4…F6, S2 = F7…F8) and globals neurals networks (one by combinaison of two subsets Sn, in my little example, it means we have 3 globals networks) , this one takes the outputs of two little NN

An illustration:

As we can see on the topic on top, we can do

fc1 = nn.Linear(784, 500)
fc2 = nn.Linear(500, 10)
optimizer = torch.optim.SGD([fc1.parameters(), fc2.parameters()], lr=0.01)

but if we do that

S_one = get_network_subset(num_subset = 0)
S_two = get_network_subset(num_subset = 1)
Global = get_network_global(nums_subsets = (0,1))
optimizer = torch.optim.SGD([S_one.parameters(), S_two.parameters(), Global.parameters()], lr=0.01)

Will Pytorch understand how to do the backpropagation (global network has the “deepest” weights) or I have to respect a specific order to pass parameters ?

At the end I want to be able to collectivize learning between models and I don’t find resources about that