So pytorch’s main strength is the ability to have dynamic computation graphs. I was wondering if there is a simple way of initializing or inserting new layers into the network as well while the network is training? For example, epoch 0 to 10 trains a single linear layer (input -> 100) and then epoch 10 to 20 adds in another layer (100 -> 1000) with random weights. Thus layer 1 is being finetuned whereas layer 2 starts training from scratch. I realize one way could be to initialize all layers one might expect to be inserted and then dynamically change the data flow. However, I wanted to dynamically initialize new layers as well. Any related documentation/code?
just create new layers on the fly behind python conditionals etc.
pytorch’s models are python code, so it’s upto you how you want to do these things. your forward function gives you full freedom to do arbitrary things.
1 Like