Appropriate ways to share weights between modules

Peyash · March 27, 2020, 7:26pm

What are the appropriate ways to share/tie parameters between modules?
For example, I have two Sequential containers containing multiple modules and I want to share/tie some of the module weights. I can’t reuse module in my case (e.g., sharing weights of Embedding with Linear)

Does setting one module weights to another do the job like this example below?

github.com

pytorch/examples/blob/master/word_language_model/model.py#L34




    # Optionally tie weights as in:
    # "Using the Output Embedding to Improve Language Models" (Press & Wolf 2016)
    # https://arxiv.org/abs/1608.05859
    # and
    # "Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling" (Inan et al. 2016)
    # https://arxiv.org/abs/1611.01462
    if tie_weights:
        if nhid != ninp:
            raise ValueError('When using the tied flag, nhid must be equal to emsize')
        self.decoder.weight = self.encoder.weight


    self.init_weights()


    self.rnn_type = rnn_type
    self.nhid = nhid
    self.nlayers = nlayers


def init_weights(self):
    initrange = 0.1
    self.encoder.weight.data.uniform_(-initrange, initrange)