# How to change the weights of a pytorch model? **URL:** \ **Category:** Uncategorized\ **Created:** [March 30, 2019, 7:43pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279 "2019-03-30T19:43:30Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [March 30, 2019, 7:43pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/1 "2019-03-30T19:43:30Z")
I need to change the weights at specific layers of ResNet-152 during training. I think there has been a similar question sometime earlier, but I cannot find it! ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 30, 2019, 8:28pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/2 "2019-03-30T20:28:00Z")
Would you like to change the weights manually? If so, you could wrap the code in a `torch.no_grad()` guard: ```python with torch.no_grad(): model.fc.weight[0, 0] = 1. ``` to prevent Autograd from tracking these changes. ---
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [March 31, 2019, 12:40pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/3 "2019-03-31T12:40:29Z")
Thank you @ptrblck! In fact, I am trying an attention mechanism and I want the weights that I update, according to attention, to take effect during training. Thus, from my understanding, `torch.no_grad()` should only be used during testing/validation. Here’s the code I have in mind (for a simple demonstration, I am replacing the attention weights here with some random values): ``` model.train(); total_loss = 0 for batch_idx, (data, target) in enumerate(train_loader): data, target = data.to(device), target.to(device) optimizer.zero_grad() attention_matrix = torch.rand(model.fc.weight.size()) model.fc.weight = model.fc.weight * attention_matrix output = model(data) loss = criterion(output, target) total_loss += loss.item() loss.backward() optimizer.step() ``` During testing, however, I think we need to use `torch.no_grad()` , as follows: ``` model.eval() with torch.no_grad(): for data, target in test_loader: data, target = data.to(device), target.to(device) model.fc.weight = model.fc.weight * attention_matrix output = model(data) loss = criterion(output, target) total_loss += loss.item() ``` I also need to do the same for **the convolution layers** , not yet sure how this could be done! ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [April 1, 2019, 12:15am UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/4 "2019-04-01T00:15:01Z")
I’m not sure if you should manipulate the weights directly using the attention weights. In the [Seq2Seq tutorial](https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder) the attention weights are multiplied with the encoder outputs to calculate new activations. Would that work for you, too? ---
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [April 1, 2019, 10:51am UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/5 "2019-04-01T10:51:11Z")
I agree with you that it is a bit critical to do this by manipulating the weights directly using attention, and the network thus might not converge. But, I intend to do this using simulated annealing, hopefully it is gonna work; I will not know unless I try it. I still do not know how to change the parameters of the convolution layers. ---
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [April 1, 2019, 5:40pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/6 "2019-04-01T17:40:02Z")
To make it work, I needed to cast the output of the multiplication, as follows: `model.fc.weight = torch.nn.parameter.Parameter( model.fc.weight * attention_matrix )` ---
**Author:** ![sudonto](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudonto/32/23727_2.png) [@sudonto](https://discuss.pytorch.org/u/sudonto)\ **Post date:** [June 29, 2020, 2:52pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/7 "2020-06-29T14:52:57Z")
@Deeply Just to make sure, so you were able to manipulate the weights manually by using the above code? No harm to model’s performance? I am asking because I want to use your approach but for conv layer. Thanks. ---
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [June 29, 2020, 5:31pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/8 "2020-06-29T17:31:11Z")
Yes. It worked. As far as I remember, the model was very tolerant to the introduced noise and was able to adapt and recover very quickly. After being hit by the first noise multiply, the MSE (or whatever metric used) was reduced a bit, but not drastically. Then, it started to recover and tolerate the noise. I was trying to generate different models at the mid-epoch training journey. Then, by having a noise generator, I aimed at building a high-level ensemble by combining all the models. **Training Configuration:** I don’t remember the exact configuration; but I can give an example about what I had in mine: suppose my training goes up to 100 epochs, the model learns normally and after epoch 50, it is hit with this noise at every 10 epochs {50, 60, 70, 80, 90} and it stops at epoch 100. Something like that. **No harm to model’s performance?** The Model performance depends on what you are trying to achieve, and how you are manipulating the weights. ---
**Author:** ![sudonto](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudonto/32/23727_2.png) [@sudonto](https://discuss.pytorch.org/u/sudonto)\ **Post date:** [June 29, 2020, 10:44pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/9 "2020-06-29T22:44:27Z")
@Deeply Thanks for your confirmation. ---
**Author:** ![akvilonBrown](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akvilonbrown/32/34909_2.png) [@akvilonBrown](https://discuss.pytorch.org/u/akvilonBrown)\ **Post date:** [March 4, 2021, 10:24am UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/10 "2021-03-04T10:24:55Z")
Greetings, I apologize for reviving this topic, it’s so close to my needs. I want to change weights according to meta-information supplied with input images and I need intentionally to track these changes with **Autograd**. I wonder if _not using_**torch.no\_grad()** is enough for it, so if I don’t use anything can I be sure that the results will be backpropagated in the usual way, and the manual alteration is compatible with “normal” gradient flow (when weights are within **nn** module). ---
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [March 4, 2021, 11:19am UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/11 "2021-03-04T11:19:19Z")
I’m not sure I understand what you are trying to achieve. Yet, IMHO, the best way to make sure that `autograd` does what it is supposed to, is to run the code in debug mode, track changes and see what has been updated via BackProbagation. I usually follow this approach to be confident about what is happening. You can run this debugging-test using a simply toy-example that mimics what you are trying to do. ---
**Author:** ![Abhishek\_Agrawal](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhishek_agrawal/32/42093_2.png) [@Abhishek\_Agrawal](https://discuss.pytorch.org/u/Abhishek_Agrawal)\ **Post date:** [September 10, 2021, 8:08pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/12 "2021-09-10T20:08:56Z")
This doesn’t seem to be the right way as in my case when I updated weights this way, my gradients became non-zero even after using the optimizer.zero\_grad(). For calculating features with updated weight, I used torch.nn.functional as we have conv layer already initialized in **init** keeping new weights in a separate variable. ---
**Author:** ![Deeply](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeply/32/3840_2.png) [@Deeply](https://discuss.pytorch.org/u/Deeply)\ **Post date:** [October 26, 2021, 2:19pm UTC](https://discuss.pytorch.org/t/how-to-change-the-weights-of-a-pytorch-model/41279/13 "2021-10-26T14:19:49Z")
In my case, I wanted the weights to be updated after hitting them with noise. Yet, you can use `torch.no_grad()` to prevent the gradient from updating your weights. You can also use detach() method, which constructs a new view on a tensor which is **declared not to need gradients** , i.e., it is to be excluded from further tracking of operations, and therefore the subgraph involving this view will not be recorded.