How to store the weights of a network as it is training?

by this i mean, after each update of the weights, i’d like to store the values in a dictionary or list. this is for a particular layer and not for all the network

Do you want to overwrite the old values after each update?
Would this work?

model = nn.Sequential(
    nn.Linear(20, 10),
    nn.ReLU(),
    nn.Linear(10, 2)
)

optimizer = optim.SGD(model.parameters(), lr=1e-3)

x = torch.randn(1, 20)

# Store just the first layer
weights = dict(model[0].named_parameters())

for epoch in range(10):
    output = model(x)
    output.mean().backward()
    optimizer.step()
    # Update weights
    weights = dict(model[0].named_parameters())

no id actually like to store all the updates, in a dictionary so that i can see how the weights change as training proceeds

Then you could try the following:

model = nn.Sequential(
    nn.Linear(20, 10),
    nn.ReLU(),
    nn.Linear(10, 2)
)

optimizer = optim.SGD(model.parameters(), lr=1e-3)

x = torch.randn(1, 20)

# Store just the first layer
weights = {k: v.clone() for k, v in model[0].named_parameters()}
updates = {k: v.clone() for k, v in model[0].named_parameters()}

for epoch in range(10):
    optimizer.zero_grad()
    output = model(x)
    output.mean().backward()
    optimizer.step()

    new_weights = {k: v.clone() for k, v in model[0].named_parameters()}
    updates = {k: new_weights[k] - weights[k] for k in weights}
    weights = new_weights
    
    print(updates)
1 Like

thank you so much! that should do it :smiley:

just for future reference, and for whoever else looks over this question. I decided to use a pickel file to store the weights as the network was being trained. It was simpler, since i needed to load the weights in another file that I was using. but thank you very much for your help