How to fix the weights and biases except two?

1

After training a model, how can I save the parameters of the network (e.g., weights and biases) and apply them to a new equal network BUT where I want to change only the first two weights?

I would like to see how the loss changes when only the first two weights varies (providing a meshgrid space) and all the others are fixed.

  1. How to fix the weights and biases except two?
  2. How to provide the domain in which the two free weights can span?
class Net(nn.Module):

    def __init__(self, layer_sizes, activation=torch.tanh):
        super(Net, self).__init__()
        self.layer_sizes = layer_sizes
        self.layer_list = [x]
        self.activation = activation

        self.l1 = nn.Linear(layer_sizes[0], layer_sizes[1])
        self.l2 = nn.Linear(layer_sizes[1], layer_sizes[2])
        self.l3 = nn.Linear(layer_sizes[2], layer_sizes[3])
        self.l4 = nn.Linear(layer_sizes[3], layer_sizes[4])

    def forward(self, x):
        x = self.l1(x)
        #print(x.dtype)
        x = self.activation(x)
        x = self.l2(x)
        x = self.activation(x)
        x = self.l3(x)
        x = self.activation(x)
        x = self.l4(x)
        return x

device         = 'cpu'
layer_sizes = [1,32,32,32,1]
activation    = torch.tanh

net_u          = Net(layer_sizes, activation).to(device)

... # training

# saving 
info2exp = []
for layer in nets[0].children():
    if isinstance(layer, nn.Linear):
        info2exp.append(layer.state_dict())

torch.save([data, epochs,  info2exp], r'./dataStorage/test_v1.pt')

How to load them to a new network? Like this?

net_new          = Net(layer_sizes, activation).to(device)
net_new.layers.load_state_dict(net_u.layers.state_dict())

print((net_new.layers[0].weight == net_u.layers[0].weight).all())
  1. How to fix the weights and biases except two?
  2. How to provide the domain in which the two free weights can span?
    Thanks!
2

Hmmm
Esentially what you save is a dicitonary.
The simplest way is changing the values in the state_dict before the new network loads them.
Something like

weights = net.state_dict()
weights['layer_name1'] = ... 
weights['layer_nameN'] = ...
new_net.load_state_dict(weights)

To freeze certain layers: How the pytorch freeze network in some layers, only the rest of the training?

1 Like
3

A1. If you want to stop training on any layer of a model, just set the requires_grad variable to False. For example:

model.l1.weight.requires_grad = False
model.l1.bias.requires_grad = False

A2. Not clear on what you’re asking.