[solved] KeyError: 'unexpected key "module.encoder.embedding.weight" in state_dict'

[Wyn_Mew]

Check out your saved model file:

check_point = torch.load('myfile.pth.tar')
check_point.key()

You may find out your ‘check_point’ got several keys such as ‘state_dict’ etc.

checkpoint = torch.load(resume)
state_dict =checkpoint['state_dict']

from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove 'module.' of dataparallel
    new_state_dict[name]=v

model.load_state_dict(new_state_dict)

[dashesy]

What about nn.DistributedDataParallel, it seems DistributedDataParallel and DataParallel can load each other’s parameters.
Is there an official way to save/load among DDP/DP/None?

[17764591637]

just do this:

model = torch.load(train_model)
…
net.load_state_dict(model[‘state_dict’])

it works for me!

[harryfyodor]

Thanks a lot, this worked for me.

[Alon]

Instead of deleting the “module.” string from all the state_dict keys, you can save your model with:
torch.save(model.module.state_dict(), path_to_file)
instead of
torch.save(model.state_dict(), path_to_file)
that way you don’t get the “module.” string to begin with…

[stoneyang]

Thanks for your hints! It saved my time

[amy_hu]

that’s work simple and perfect for me! thanks

[KaiyangZhou (Kaiyang)]

In case someone needs, this function can handle loading weights w/ and w/o ‘module’.

To save model without ‘module’, you may try this.

[weblucas (Lucas Teixeira)]

After pytorch 1.xx
this was fixed, now you only need to do this

            if isinstance(args.pretrained, torch.nn.DataParallel):
                args.pretrained = args.pretrained.module

[achaiah]

@weblucas

What is args in this case? Is it project-specific?

[weblucas (Lucas Teixeira)]

This code that I am using saves the model using torch.save(model)… in this case the model is load using args.pretrained = torch.load(args.pretrained)
when it is a single gpu. model is one of my models MyModelNet(nn.Module), but in the multi gpu case it is nn.DataParallel(MyModelNet(nn.Module))

[achaiah]

Ok so that wouldn’t really fix the loading problem but will help saving the correct state_dict() depending on whether your model is parallelized or not.

[Quisher (Tallal Javed)]

A more graceful solution is:

name = k.replace(".module", “”) # removing ‘.moldule’ from key

As for me using the k[7:] wasn’t properly removing the ‘module’.

[vorab]

@fmassa

I used your code to remove unexpected keys, however I can not get out from this error.
I tried all other tricks also.

Please give some hints to solve it. It will be very appreciated.

state_dict = torch.load("/media/Data/jcl-vb/output_dir/model_0000050.pth")
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
name = k[9:] # remove module.
new_state_dict[name] = v
model.load_state_dict(new_state_dict)
‘’

However, it shows the error like:

RuntimeError: Error(s) in loading state_dict for GeneralizedRCNN:
Missing key(s) in state_dict: “backbone.body.stem.conv1.weight”, “backbone.body.stem.bn1.weight”, “backbone.body.stem.bn1.bias”, “backbone.body.stem.bn1.running_mean”, “backbone.body.stem.bn1.running_var”, “backbone.body.layer1.0.downsample.0.weight”, “backbone.body.layer1.0.downsample.1.weight”, “backbone.body.layer1.0.downsample.1.bias”, “backbone.body.layer1.0.downsample.1.running_mean”, “backbone.body.layer1.0.downsample.1.running_var”, “backbone.body.layer1.0.conv1.weight”, “backbone.body.layer1.0.bn1.weight”, “backbone.body.layer1.0.bn1.bias”, “backbone.body.layer1.0.bn1.running_mean”, “backbone.body.layer1.0.bn1.running_var”, “backbone.body.layer1.0.conv2.weight”, “backbone.body.layer1.0.bn2.weight”, “backbone.body.layer1.0.bn2.bias”, “backbone.body.layer1.0.bn2.running_mean”, “backbone.body.layer1.0.bn2.running_var”, “backbone.body.layer1.0.conv3.weight”, “backbone.body.layer1.0.bn3.weight”, “backbone.body.layer1.0.bn3.bias”, “backbone.body.layer1.0.bn3.running_mean”, “backbone.body.layer1.0.bn3.running_var”, “backbone.body.layer1.1.conv1.weight”, “backbone.body.layer1.1.bn1.weight”, “backbone.body.layer1.1.bn1.bias”, “backbone.body.layer1.1.bn1.running_mean”, “backbone.body.layer1.1.bn1.running_var”, “backbone.body.layer1.1.conv2.weight”, “backbone.body.layer1.1.bn2.weight”, “backbone.body.layer1.1.bn2.bias”, “backbone.body.layer1.1.bn2.running_mean”, “backbone.body.layer1.1.bn2.running_var”, “backbone.body.layer1.1.conv3.weight”, “backbone.body.layer1.1.bn3.weight”, “backbone.body.layer1.1.bn3.bias”, “backbone.body.layer1.1.bn3.running_mean”, “backbone.body.layer1.1.bn3.running_var”, “backbone.body.layer1.2.conv1.weight”, “backbone.body.layer1.2.bn1.weight”, “backbone.body.layer1.2.bn1.bias”, “backbone.body.layer1.2.bn1.running_mean”, “backbone.body.layer1.2.bn1.running_var”, “backbone.body.layer1.2.conv2.weight”, “backbone.body.layer1.2.bn2.weight”, “backbone.body.layer1.2.bn2.bias”, “backbone.body.layer1.2.bn2.running_mean”, “backbone.body.layer1.2.bn2.running_var”, “backbone.body.layer1.2.conv3.weight”, “backbone.body.layer1.2.bn3.weight”, “backbone.body.layer1.2.bn3.bias”, “backbone.body.layer1.2.bn3.running_mean”, “backbone.body.layer1.2.bn3.running_var”, “backbone.body.layer2.0.downsample.0.weight”, “backbone.body.layer2.0.downsample.1.weight”, “backbone.body.layer2.0.downsample.1.bias”, “backbone.body.layer2.0.downsample.1.running_mean”, “backbone.body.layer2.0.downsample.1.running_var”, “backbone.body.layer2.0.conv1.weight”, “backbone.body.layer2.0.bn1.weight”, “backbone.body.layer2.0.bn1.bias”, “backbone.body.layer2.0.bn1.running_mean”, “backbone.body.layer2.0.bn1.running_var”, “backbone.body.layer2.0.conv2.weight”, “backbone.body.layer2.0.bn2.weight”, “backbone.body.layer2.0.bn2.bias”, “backbone.body.layer2.0.bn2.running_mean”, “backbone.body.layer2.0.bn2.running_var”, “backbone.body.layer2.0.conv3.weight”, “backbone.body.layer2.0.bn3.weight”, “backbone.body.layer2.0.bn3.bias”, “backbone.body.layer2.0.bn3.running_mean”, “backbone.body.layer2.0.bn3.running_var”, “backbone.body.layer2.1.conv1.weight”, “backbone.body.layer2.1.bn1.weight”, “backbone.body.layer2.1.bn1.bias”, “backbone.body.layer2.1.bn1.running_mean”, “backbone.body.layer2.1.bn1.running_var”, “backbone.body.layer2.1.conv2.weight”, “backbone.body.layer2.1.bn2.weight”, “backbone.body.layer2.1.bn2.bias”, “backbone.body.layer2.1.bn2.running_mean”, “backbone.body.layer2.1.bn2.running_var”, “backbone.body.layer2.1.conv3.weight”, “backbone.body.layer2.1.bn3.weight”, “backbone.body.layer2.1.bn3.bias”, “backbone.body.layer2.1.bn3.running_mean”, “backbone.body.layer2.1.bn3.running_var”, “backbone.body.layer2.2.conv1.weight”, “backbone.body.layer2.2.bn1.weight”, “backbone.body.layer2.2.bn1.bias”, “backbone.body.layer2.2.bn1.running_mean”, “backbone.body.layer2.2.bn1.running_var”, “backbone.body.layer2.2.conv2.weight”, “backbone.body.layer2.2.bn2.weight”, “backbone.body.layer2.2.bn2.bias”, “backbone.body.layer2.2.bn2.running_mean”, “backbone.body.layer2.2.bn2.running_var”, “backbone.body.layer2.2.conv3.weight”, “backbone.body.layer2.2.bn3.weight”, “backbone.body.layer2.2.bn3.bias”, “backbone.body.layer2.2.bn3.running_mean”, “backbone.body.layer2.2.bn3.running_var”, “backbone.body.layer2.3.conv1.weight”, “backbone.body.layer2.3.bn1.weight”, “backbone.body.layer2.3.bn1.bias”, “backbone.body.layer2.3.bn1.running_mean”, “backbone.body.layer2.3.bn1.running_var”, “backbone.body.layer2.3.conv2.weight”, “backbone.body.layer2.3.bn2.weight”, “backbone.body.layer2.3.bn2.bias”, “backbone.body.layer2.3.bn2.running_mean”, “backbone.body.layer2.3.bn2.running_var”, “backbone.body.layer2.3.conv3.weight”, “backbone.body.layer2.3.bn3.weight”, “backbone.body.layer2.3.bn3.bias”, “backbone.body.layer2.3.bn3.running_mean”, “backbone.body.layer2.3.bn3.running_var”, “backbone.body.layer3.0.downsample.0.weight”, “backbone.body.layer3.0.downsample.1.weight”, “backbone.body.layer3.0.downsample.1.bias”, “backbone.body.layer3.0.downsample.1.running_mean”, “backbone.body.layer3.0.downsample.1.running_var”, “backbone.body.layer3.0.conv1.weight”, “backbone.body.layer3.0.bn1.weight”, “backbone.body.layer3.0.bn1.bias”, “backbone.body.layer3.0.bn1.running_mean”, “backbone.body.layer3.0.bn1.running_var”, “backbone.body.layer3.0.conv2.weight”, “backbone.body.layer3.0.bn2.weight”, “backbone.body.layer3.0.bn2.bias”, “backbone.body.layer3.0.bn2.running_mean”, “backbone.body.layer3.0.bn2.running_var”, “backbone.body.layer3.0.conv3.weight”, “backbone.body.layer3.0.bn3.weight”, “backbone.body.layer3.0.bn3.bias”, “backbone.body.layer3.0.bn3.running_mean”, “backbone.body.layer3.0.bn3.running_var”, “backbone.body.layer3.1.conv1.weight”, “backbone.body.layer3.1.bn1.weight”, “backbone.body.layer3.1.bn1.bias”, “backbone.body.layer3.1.bn1.running_mean”, “backbone.body.layer3.1.bn1.running_var”, “backbone.body.layer3.1.conv2.weight”, “backbone.body.layer3.1.bn2.weight”, “backbone.body.layer3.1.bn2.bias”, “backbone.body.layer3.1.bn2.running_mean”, “backbone.body.layer3.1.bn2.running_var”, “backbone.body.layer3.1.conv3.weight”, “backbone.body.layer3.1.bn3.weight”, “backbone.body.layer3.1.bn3.bias”, “backbone.body.layer3.1.bn3.running_mean”, “backbone.body.layer3.1.bn3.running_var”, “backbone.body.layer3.2.conv1.weight”, “backbone.body.layer3.2.bn1.weight”, “backbone.body.layer3.2.bn1.bias”, “backbone.body.layer3.2.bn1.running_mean”, “backbone.body.layer3.2.bn1.running_var”, “backbone.body.layer3.2.conv2.weight”, “backbone.body.layer3.2.bn2.weight”, “backbone.body.layer3.2.bn2.bias”, “backbone.body.layer3.2.bn2.running_mean”, “backbone.body.layer3.2.bn2.running_var”, “backbone.body.layer3.2.conv3.weight”, “backbone.body.layer3.2.bn3.weight”, “backbone.body.layer3.2.bn3.bias”, “backbone.body.layer3.2.bn3.running_mean”, “backbone.body.layer3.2.bn3.running_var”, “backbone.body.layer3.3.conv1.weight”, “backbone.body.layer3.3.bn1.weight”, “backbone.body.layer3.3.bn1.bias”, “backbone.body.layer3.3.bn1.running_mean”, “backbone.body.layer3.3.bn1.running_var”, “backbone.body.layer3.3.conv2.weight”, “backbone.body.layer3.3.bn2.weight”, “backbone.body.layer3.3.bn2.bias”, “backbone.body.layer3.3.bn2.running_mean”, “backbone.body.layer3.3.bn2.running_var”, “backbone.body.layer3.3.conv3.weight”, “backbone.body.layer3.3.bn3.weight”, “backbone.body.layer3.3.bn3.bias”, “backbone.body.layer3.3.bn3.running_mean”, “backbone.body.layer3.3.bn3.running_var”, “backbone.body.layer3.4.conv1.weight”, “backbone.body.layer3.4.bn1.weight”, “backbone.body.layer3.4.bn1.bias”, “backbone.body.layer3.4.bn1.running_mean”, “backbone.body.layer3.4.bn1.running_var”, “backbone.body.layer3.4.conv2.weight”, “backbone.body.layer3.4.bn2.weight”, “backbone.body.layer3.4.bn2.bias”, “backbone.body.layer3.4.bn2.running_mean”, “backbone.body.layer3.4.bn2.running_var”, “backbone.body.layer3.4.conv3.weight”, “backbone.body.layer3.4.bn3.weight”, “backbone.body.layer3.4.bn3.bias”, “backbone.body.layer3.4.bn3.running_mean”, “backbone.body.layer3.4.bn3.running_var”, “backbone.body.layer3.5.conv1.weight”, “backbone.body.layer3.5.bn1.weight”, “backbone.body.layer3.5.bn1.bias”, “backbone.body.layer3.5.bn1.running_mean”, “backbone.body.layer3.5.bn1.running_var”, “backbone.body.layer3.5.conv2.weight”, “backbone.body.layer3.5.bn2.weight”, “backbone.body.layer3.5.bn2.bias”, “backbone.body.layer3.5.bn2.running_mean”, “backbone.body.layer3.5.bn2.running_var”, “backbone.body.layer3.5.conv3.weight”, “backbone.body.layer3.5.bn3.weight”, “backbone.body.layer3.5.bn3.bias”, “backbone.body.layer3.5.bn3.running_mean”, “backbone.body.layer3.5.bn3.running_var”, “backbone.body.layer4.0.downsample.0.weight”, “backbone.body.layer4.0.downsample.1.weight”, “backbone.body.layer4.0.downsample.1.bias”, “backbone.body.layer4.0.downsample.1.running_mean”, “backbone.body.layer4.0.downsample.1.running_var”, “backbone.body.layer4.0.conv1.weight”, “backbone.body.layer4.0.bn1.weight”, “backbone.body.layer4.0.bn1.bias”, “backbone.body.layer4.0.bn1.running_mean”, “backbone.body.layer4.0.bn1.running_var”, “backbone.body.layer4.0.conv2.weight”, “backbone.body.layer4.0.bn2.weight”, “backbone.body.layer4.0.bn2.bias”, “backbone.body.layer4.0.bn2.running_mean”, “backbone.body.layer4.0.bn2.running_var”, “backbone.body.layer4.0.conv3.weight”, “backbone.body.layer4.0.bn3.weight”, “backbone.body.layer4.0.bn3.bias”, “backbone.body.layer4.0.bn3.running_mean”, “backbone.body.layer4.0.bn3.running_var”, “backbone.body.layer4.1.conv1.weight”, “backbone.body.layer4.1.bn1.weight”, “backbone.body.layer4.1.bn1.bias”, “backbone.body.layer4.1.bn1.running_mean”, “backbone.body.layer4.1.bn1.running_var”, “backbone.body.layer4.1.conv2.weight”, “backbone.body.layer4.1.bn2.weight”, “backbone.body.layer4.1.bn2.bias”, “backbone.body.layer4.1.bn2.running_mean”, “backbone.body.layer4.1.bn2.running_var”, “backbone.body.layer4.1.conv3.weight”, “backbone.body.layer4.1.bn3.weight”, “backbone.body.layer4.1.bn3.bias”, “backbone.body.layer4.1.bn3.running_mean”, “backbone.body.layer4.1.bn3.running_var”, “backbone.body.layer4.2.conv1.weight”, “backbone.body.layer4.2.bn1.weight”, “backbone.body.layer4.2.bn1.bias”, “backbone.body.layer4.2.bn1.running_mean”, “backbone.body.layer4.2.bn1.running_var”, “backbone.body.layer4.2.conv2.weight”, “backbone.body.layer4.2.bn2.weight”, “backbone.body.layer4.2.bn2.bias”, “backbone.body.layer4.2.bn2.running_mean”, “backbone.body.layer4.2.bn2.running_var”, “backbone.body.layer4.2.conv3.weight”, “backbone.body.layer4.2.bn3.weight”, “backbone.body.layer4.2.bn3.bias”, “backbone.body.layer4.2.bn3.running_mean”, “backbone.body.layer4.2.bn3.running_var”, “backbone.fpn.fpn_inner1.weight”, “backbone.fpn.fpn_inner1.bias”, “backbone.fpn.fpn_layer1.weight”, “backbone.fpn.fpn_layer1.bias”, “backbone.fpn.fpn_inner2.weight”, “backbone.fpn.fpn_inner2.bias”, “backbone.fpn.fpn_layer2.weight”, “backbone.fpn.fpn_layer2.bias”, “backbone.fpn.fpn_inner3.weight”, “backbone.fpn.fpn_inner3.bias”, “backbone.fpn.fpn_layer3.weight”, “backbone.fpn.fpn_layer3.bias”, “backbone.fpn.fpn_inner4.weight”, “backbone.fpn.fpn_inner4.bias”, “backbone.fpn.fpn_layer4.weight”, “backbone.fpn.fpn_layer4.bias”, “rpn.anchor_generator.cell_anchors.0”, “rpn.anchor_generator.cell_anchors.1”, “rpn.anchor_generator.cell_anchors.2”, “rpn.anchor_generator.cell_anchors.3”, “rpn.anchor_generator.cell_anchors.4”, “rpn.head.conv.weight”, “rpn.head.conv.bias”, “rpn.head.cls_logits.weight”, “rpn.head.cls_logits.bias”, “rpn.head.bbox_pred.weight”, “rpn.head.bbox_pred.bias”, “roi_heads.box.feature_extractor.fc6.weight”, “roi_heads.box.feature_extractor.fc6.bias”, “roi_heads.box.feature_extractor.fc7.weight”, “roi_heads.box.feature_extractor.fc7.bias”, “roi_heads.box.predictor.cls_score.weight”, “roi_heads.box.predictor.cls_score.bias”, “roi_heads.box.predictor.bbox_pred.weight”, “roi_heads.box.predictor.bbox_pred.bias”.
Unexpected key(s) in state_dict: “”.

[Tavados]

Yup, same here. In my case it’s e.g. “synth_reader.0.weight” and “synth_reader.module.0.weight”, so the replace works like a charm.

[lelegu]

Can you explain more clearly how to add a nn.DataParallel temporarily in your network for loading purposes? e.g. can you provide a simple example?
I am new to pytorch, thanks so much!

[Kanchan_Sarkar]

you can use strict=False in load_state_dict. This can solved the issue.

model.load_state_dict(checkpoint['state_dict'], strict=False)

[arturs.polis]

I would recommend caution in using strict=False here. I tried this, and replacing module. worked correctly and gave me the validation results as expected from my model, while strict=False gave wrong results (even though PyTorch did not complain).

[kingtheon]

It worked! Huuuuge thanks.

[marbakken]

As @arturs.polis also mentioned, be careful with this! For me, it removed the error message, but gave a completely wrong model without warning!