# Averaging some model parameters

**URL:** <https://discuss.pytorch.org/t/averaging-some-model-parameters/135925>\
**Category:** autograd\
**Created:** [November 4, 2021, 3:05am UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925 "2021-11-04T03:05:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Evan\_Vogelbaum](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evan_vogelbaum/32/13250_2.png) [@Evan\_Vogelbaum](https://discuss.pytorch.org/u/Evan_Vogelbaum)\
**Post date:** [November 4, 2021, 3:05am UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925/1 "2021-11-04T03:05:33Z")

</div>

I have k classes, for each of which I have trained a model. I would like to create a new model whose architecture is identical and whose parameters are the **weighted** average of the corresponding parameters of each of my k models according to some specified distribution p (e.g. [0.1, 0.2, 0.7] for k=3).

My goal is to construct such a model, evaluate it on data, and then have gradient propagate back to the original models (the intermediate averaged model does not matter to me). Would someone be able to help me understand the best way to do this?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 4, 2021, 6:18am UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925/2 "2021-11-04T06:18:49Z")

</div>

To simply average the parameters of multiple models you could use [this approach](https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008/2). However, based on your description you would like to calculate the gradients for the parameters in the original models, so using the averaged `state_dict` wouldn’t work.  
Instead I think the right approach would be to create the new averaged tensors from all models and use the functional API for these new tensors.  
Something like this would work:

```python
model1 = nn.Linear(10, 10)
model2 = nn.Linear(10, 10)

new_weight = (model1.weight + model2.weight) / 2.
new_bias = (model1.bias + model2.bias) / 2.

x = torch.randn(1, 10)
out = F.linear(x, new_weight, new_bias)

out.mean().backward()

for name, param in model1.named_parameters():
    print(name, param.grad)
    
for name, param in model2.named_parameters():
    print(name, param.grad)

```

Although it seems to be quite cumbersome, so maybe someone else would have a better and cleaner approach.

---

<div class="post-metadata">

**Author:** ![Evan\_Vogelbaum](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evan_vogelbaum/32/13250_2.png) [@Evan\_Vogelbaum](https://discuss.pytorch.org/u/Evan_Vogelbaum)\
**Post date:** [November 4, 2021, 1:30pm UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925/3 "2021-11-04T13:30:21Z")

</div>

Thanks so much! This certainly works for very simple models but I agree is extremely cumbersome for larger ones. Do you know why it is that `nn.Parameter`s don’t retain the gradient of the tensors which are used to create then?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 5, 2021, 7:17am UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925/4 "2021-11-05T07:17:57Z")

</div>

`nn.Parameter` would recreate a new parameter without any gradient history. While this new parameter would then get the gradients the two tensors used to create it wouldn’t.

---

<div class="post-metadata">

**Author:** ![Evan\_Vogelbaum](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evan_vogelbaum/32/13250_2.png) [@Evan\_Vogelbaum](https://discuss.pytorch.org/u/Evan_Vogelbaum)\
**Post date:** [November 5, 2021, 1:48pm UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925/5 "2021-11-05T13:48:41Z")

</div>

Yeah; a bit of searching around showed me that [this is still an open issue](https://discuss.pytorch.org/t/nn-parameter-doesnt-retain-grad-fn/29214). Hopefully this is a feature added soon. For now the thread is closed. Thanks again @ptrblck !

---

<div class="post-metadata">

**Author:** ![borgr](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/borgr/32/44213_2.png) [@borgr](https://discuss.pytorch.org/u/borgr)\
**Post date:** [March 2, 2022, 12:41pm UTC](https://discuss.pytorch.org/t/averaging-some-model-parameters/135925/6 "2022-03-02T12:41:20Z")

</div>

If there are “real” models, this approach is infeasible (e.g. there is no “attention” functional and you need to reimplement everything anew). Is there another way someone figured for that?