# Re-using layers in model?

**URL:** <https://discuss.pytorch.org/t/re-using-layers-in-model/48186>\
**Category:** Uncategorized\
**Created:** [June 17, 2019, 7:03pm UTC](https://discuss.pytorch.org/t/re-using-layers-in-model/48186 "2019-06-17T19:03:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajhanwar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajhanwar/32/13182_2.png) [@ajhanwar](https://discuss.pytorch.org/u/ajhanwar)\
**Post date:** [June 17, 2019, 7:03pm UTC](https://discuss.pytorch.org/t/re-using-layers-in-model/48186/1 "2019-06-17T19:03:35Z")

</div>

Hi all, I am learning about PyTorch and have been practicing through implementing the YOLOv3 model.

The model itself has layers which are “repeated” in a sense. For example, we repeat the process of doing the following several times:

```auto
nn.Conv2d(256, 512, kernel_size=1, stride=1, padding=1)
nn.Conv2d(512, 256, kernel_size=3, stride=1, padding=1)
nn.Conv2d(256, 512, kernel_size=1, stride=1, padding=1)
nn.Conv2d(512, 256, kernel_size=3, stride=1, padding=1)
...

```

In such a situation would we need to explicitly list out each convolutional layer within the model definition? I understand each layer has an associated weight tensor and so I am not sure if we can simply re-use the layers.

---

<div class="post-metadata">

**Author:** ![tymokvo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tymokvo/32/9527_2.png) [@tymokvo](https://discuss.pytorch.org/u/tymokvo)\
**Post date:** [June 17, 2019, 7:32pm UTC](https://discuss.pytorch.org/t/re-using-layers-in-model/48186/2 "2019-06-17T19:32:05Z")

</div>

Welcome to the forums!

Yes it is definitely possible! Whether or not you want to do it is a design decision. Broadly, it will lower memory cost while also reducing the degrees of freedom of the model.  
One of the important contributions of [DenseNet](https://arxiv.org/abs/1608.06993) was the discovery that (for classification) you can pass the parameter tensors of early layers as inputs to deeper layers to reduce the memory cost of the model and encourage the model to learn features that are applicable at several scales.

You could share the parameters themselves like:

```auto
c = nn.Conv2d(3, 3, (3, 3))

im = torch.randn(1, 3, 8, 8, requires_grad=True)

o = c(im)
o = c(o)

```

And a render of its backprop graph looks like:

 ![image](https://discuss.pytorch.org/uploads/default/original/2X/d/d227844c23e50fc970779c8fc76c31d6903192ce.png)

Or a DenseNet idea like:

```auto
c = nn.Conv2d(3, 3, (3, 3), 1, 1, bias=False)
u = nn.UpsamplingBilinear2d(scale_factor=3)

im = torch.randn(1, 3, 9, 9, requires_grad=True)
fms = u(c.weight)

o = c(im)
o = torch.cat([fms, o])

```

Which looks like:

 ![image](https://discuss.pytorch.org/uploads/default/original/2X/4/4dc405e705d2580a12a832cc169534df40d015f7.png)

I am not sure if YOLO uses parameter sharing. I do seem to remember that YOLOv3 does use residual connections. But it would be an interesting experiment to see if the parameter sharing improves performance!

---

<div class="post-metadata">

**Author:** ![ajhanwar](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajhanwar/32/13182_2.png) [@ajhanwar](https://discuss.pytorch.org/u/ajhanwar)\
**Post date:** [June 17, 2019, 8:35pm UTC](https://discuss.pytorch.org/t/re-using-layers-in-model/48186/3 "2019-06-17T20:35:46Z")

</div>

Thank you so much for the detailed explanation!  
Just to confirm my understanding, reusing layers would effectively count as parameter sharing (using same weights)? And if so, is there a way to prevent such an occurrence without having to explicitly list out every layer?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 17, 2019, 10:51pm UTC](https://discuss.pytorch.org/t/re-using-layers-in-model/48186/4 "2019-06-17T22:51:42Z")

</div>

Yes, reusing a layer would use the same parameters in its operation.  
If you are dealing with a lot of (simple to initialize) layers, you could try to create them in a loop and append each one to a `nn.ModuleList`.  
Later on you could pass the layers from this list to an `nn.Sequential` module or just call it again in a loop in your `forward` method.