# Parameter / Weight sharing

**URL:** <https://discuss.pytorch.org/t/parameter-weight-sharing/1318>\
**Category:** Uncategorized\
**Created:** [March 24, 2017, 6:57am UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318 "2017-03-24T06:57:14Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kentsommer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kentsommer/32/338_2.png) [@kentsommer](https://discuss.pytorch.org/u/kentsommer)\
**Post date:** [March 24, 2017, 6:57am UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/1 "2017-03-24T06:57:14Z")

</div>

I’m a little lost on how it would be possible to perform weight sharing in pytorch. In my case I would like to do the following:

Essentially I would like to reuse weights q.data.weights and weights w2 concatenated together in a loop. q.data.weights are the weights for conv2d layer q as well as being used in the loop, while weights w2 would ONLY be used in the conv2d layer in the loop (however, since it would be repeated n times, technically weights w2 would be shared as well).

```auto
class Model(nn.Module):

    def __init__ (self, config):
        super(Model, self). __init__ ()
        self.config = config
        self.w2 = Variable(torch.randn(10, 1, 3, 3), requires_grad=True)
        self.h = nn.Conv2d(in_channels=2, 
                           out_channels=150, 
                           kernel_size=(3, 3), 
                           stride=1, padding=1) 
        self.r = nn.Conv2d(in_channels=150, 
                           out_channels=1, 
                           kernel_size=(1, 1), 
                           stride=1, padding=1)
        self.q = nn.Conv2d(in_channels=1, 
                           out_channels=10, 
                           kernel_size=(3, 3), 
                           stride=1, padding=1) 

    def forward(self, x):
        h = self.h(x)
        r = self.r(h)
        q = self.q(r)
        v = torch.max(q, dim=1)[0]
        for i in range(n):
            rv = torch.cat([r, v], 1)
            Conv2d(rv, weights=[self.q.weight, self.w2]) # This is obviously not valid torch code

```

In Theano it could be accomplished as follows:

```auto
# Helper function
def conv2D_keep_shape(x, w, subsample=(1, 1)):
    # crop output to same size as input
    fs = T.shape(w)[2] - 1 # this is the filter size minus 1
    ims = T.shape(x)[2] # this is the image size
    return theano.sandbox.cuda.dnn.dnn_conv(img=x,
                                            kerns=w,
                                            border_mode='full',
                                            subsample=subsample,
                                            )[:, :, fs/2:ims+fs/2, fs/2:ims+fs/2]

# Weights
self.w = theano.shared((np.random.randn(1, 150, 1, 1)).astype(theano.config.floatX))
self.w1 = theano.shared((np.random.randn(10, 1, 3, 3)).astype(theano.config.floatX))
self.w2 = theano.shared((np.random.randn(10, 1, 3, 3)).astype(theano.config.floatX))

# Model
self.r = conv2D_keep_shape(input, self.w)

self.q = conv2D_keep_shape(self.r, self.w1)

self.v = T.max(self.q, axis=1, keepdims=True)

for i in range(n):
    self.q = conv2D_keep_shape(T.concatenate([self.r, self.v], axis=1), T.concatenate([self.w1, self.w2], axis=1))

```

Is this possible to achieve in pytorch? Please also let me know if you need more explination in case the above is not clear.

---

<div class="post-metadata">

**Author:** ![kentsommer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kentsommer/32/338_2.png) [@kentsommer](https://discuss.pytorch.org/u/kentsommer)\
**Post date:** [March 24, 2017, 8:03am UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/2 "2017-03-24T08:03:05Z")

</div>

Just thought of a possible workaround although its fairly hacky and I’m hoping there is a cleaner way to do it.

It would (at least in theory), be possible to modify the Conv2d class in `/torch/nn/modules/conv.py` so that the `forward` function could accept “weight” as a parameter if supplied, else simply use `self.weight`

Not super jazzed about needing to modify the pytorch source unless its really necessary. So hopefully someone with more than my admittedly short (one day) experience with pytorch will have a better solution.

---

<div class="post-metadata">

**Author:** ![JerryLin](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/34f0e0/32.png) [@JerryLin](https://discuss.pytorch.org/u/JerryLin)\
**Post date:** [March 24, 2017, 8:12am UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/3 "2017-03-24T08:12:59Z")

</div>

You can use the functional form of `Conv2d` ([http://pytorch.org/docs/nn.html#id14](http://pytorch.org/docs/nn.html#id14)).

---

<div class="post-metadata">

**Author:** ![kentsommer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kentsommer/32/338_2.png) [@kentsommer](https://discuss.pytorch.org/u/kentsommer)\
**Post date:** [March 24, 2017, 8:31am UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/4 "2017-03-24T08:31:10Z")

</div>

So would this then be the correct way to do this? (is dropping a functional component into the forward() function valid?)

```auto

    def forward(self, x):
        h = self.h(x)
        r = self.r(h)
        q = self.q(r)
        v = torch.max(q, dim=1)[0]
        for i in range(n):
            q = F.conv2d(torch.cat([r, v], 1), 
                         torch.cat([self.q.weight, self.w_fb], 1), 
                         padding=1)
            v = torch.max(q, dim=1)[0]

```

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [March 24, 2017, 3:52pm UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/5 "2017-03-24T15:52:02Z")

</div>

yes that’s a way to do it.  
And yes, dropping a functional component in `forward()` is valid.

---

<div class="post-metadata">

**Author:** ![Prasanna1991](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/32.png) [@Prasanna1991](https://discuss.pytorch.org/u/Prasanna1991)\
**Post date:** [July 23, 2017, 5:38pm UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/6 "2017-07-23T17:38:03Z")

</div>

Hi,

I want to know the answer from Torch perceptive. If my network looks like this:  
a = nn.Sequential()  
a:add(nn.Linear(10,20))  
a:add(nn.Linear(10,20))

I would have done:  
a:get(1).weight:set(a:get(2).weight) Similar for bias, gradWeight, and gradBias

In PyTorch:  
self.fc1 = nn.Linear(10,20)  
self.fc7 = nn.Linear(10,20)

What should I do?  
self.fc7.weight = self.fc1.weight and similar for bias?  
As I see there is no set command. And also linear object has no attribute for ‘gradWeight’!

Would that be enough to share gradWeights and gradBias?

Thanks.

---

<div class="post-metadata">

**Author:** ![Baichuan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/baichuan/32/21016_2.png) [@Baichuan](https://discuss.pytorch.org/u/Baichuan)\
**Post date:** [February 27, 2020, 9:14am UTC](https://discuss.pytorch.org/t/parameter-weight-sharing/1318/7 "2020-02-27T09:14:29Z")

</div>

Do you mean something like [How to create model with sharing weight?](https://discuss.pytorch.org/t/how-to-create-model-with-sharing-weight/398/2)?

```auto
for i in range(n):
q=q(torch.cat([r,v]))

```