Hello everyone! I’m trying to find a way to train a linear layer (vector) that will optimize the way a number of pytorch models are combined, but I’m not sure where to start.
The idea would be something like: having 4 independently trained models (same network, shapes and number of weights), a linear layer that has a weight that corresponds to each input model and what I would like to do is having the resulting combination evaluated with a training dataset and backpropagate the error to modify only the linear layer, so it can be optimized given a loss function.
It’s a bit similar to the question asked in Combining Trained Models in PyTorch, but in my case I’m working with CNN networks, and instead would like to optimize how to combine the models instead of combining the results of the models.
Is there a way this is feasible?