Understanding tied weights

Jay_Timbadia · March 21, 2022, 8:17am

Hey,

I was coding some models using pytorch and came across this

model.set_tied()

I did searched for the documentation on pytorch, but didn’t found anything.
Can you please let me know what does this line of code does.
If there is any brief resource, can you please point me out?

github.com

graykode/gpt-2-Pytorch/blob/master/GPT2/utils.py

'''
    code by TaeHwan Jung(@graykode)
    Original Paper and repository here : https://github.com/openai/gpt-2
    GPT2 Pytorch Model : https://github.com/huggingface/pytorch-pretrained-BERT
'''
import logging

logger = logging.getLogger(__name__)

def load_weight(model, state_dict):
    old_keys = []
    new_keys = []
    for key in state_dict.keys():
        new_key = None
        if key.endswith(".g"):
            new_key = key[:-2] + ".weight"
        elif key.endswith(".b"):
            new_key = key[:-2] + ".bias"
        elif key.endswith(".w"):
            new_key = key[:-2] + ".weight"

This file has been truncated. show original

Thank you!

ptrblck · March 22, 2022, 6:05am

set_tied is not a built-in nn.Module method and defined in the repository here.