Suppose I have a network that looks like
layer_1 -> layer_2 -> layer_3. I have all the weights pretrained. Suppose
layer_1's output is
layer_2's output is
Now I want to train another layer,
my_layer_2, with the original
output_1 as input and
output_2 as output, so that if I replace
layer_2 in the original network with a trained
my_layer_2, the difference won’t be large.
What is a good way to do this? Is there any existing mechanism, or do I need to somehow hack it myself?