Question About Adding Parameters/Biases

[J_Johnson]

Hello!

If I want to add more neurons to a layer inside of a model during training, do I only need to update the model.parameters() for each layer(weights and biases) after backpropagation, or is there more to it than that? If so, what would I need to do in addition to resizing the parameter tensors?

[ptrblck]

Since you are changing the parameters, you would need to pass these new parameters to the optimizer (or create a new one). The running stats in this optimizer would be lost in this case (if it’s using internal estimates).

[J_Johnson]

After reading several posts, I’ve come to this same conclusion. Which leads to my next question, here:

Modifying/Custom Optimizer: Updating Layer Parameters

I’m working on a modification to a pytorch optimizer file. For simplicity sake, let’s take SGD(sgd.py). If I wanted to change the “for loop” on line 94 to: for i, p in enumerate(group['params']: And add something starting on line 111 in the for loop like: if i==0: bias=torch.rand(10) group['params'][i+1]=bias Also, suppose that the number of biases are 8 in this layer before going through this ‘optimizer’. Currently, it doesn’t update the parameters when I run this. What would I need to …