Why do we need state_dict = state_dict.copy()

Tec_dan · April 30, 2020, 4:28pm

Hello everyone,

I haven’t used PyTorch for a long time and there is one line code confused me very much. If any of you can help explain, I will appreciate it very much.

In my task I want to load the weights from the already pretrained model to my local model. But I don’t understand why we need this line " state_dict = state_dict.copy() " if the two have the same name " state_dict".

    # copy state_dict so _load_from_state_dict can modify it
    metadata = getattr(state_dict, '_metadata', None)
    state_dict = state_dict.copy()
    if metadata is not None:
        state_dict._metadata = metadata

    def load(module, prefix=''):

        local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
        module._load_from_state_dict(
            state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
        for name, child in module._modules.items():
            if child is not None:
                load(child, prefix + name + '.')
    start_prefix = ''
    # print("hasattr(model, 'bert')",hasattr(model, 'bert')  ) :false
    if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
        start_prefix = 'bert.'
    load(model, prefix=start_prefix)

Just like how Huggingface use it in the following link(line: 669).

github.com

huggingface/transformers/blob/master/src/transformers/modeling_utils.py

# coding=utf-8
# Copyright 2018 The Google AI Language Team Authors, Facebook AI Research authors and The HuggingFace Inc. team.
# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""PyTorch BERT model."""

import logging
import os
from typing import Callable, Tuple

This file has been truncated. show original

wtliao · October 6, 2021, 1:22pm

I am also confused about this line. Have you figured it out? I got nothing after a whole day google

BramVanroy · October 6, 2021, 2:12pm

This might be helpful: Cache = self.state_dict() overwritten - #6 by ptrblck