# \[solved\] KeyError: 'unexpected key "module.encoder.embedding.weight" in state\_dict'

**URL:** <https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686>\
**Category:** Uncategorized\
**Created:** [April 6, 2017, 8:11am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686 "2017-04-06T08:11:10Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![wasiahmad](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasiahmad/32/248_2.png) [@wasiahmad](https://discuss.pytorch.org/u/wasiahmad)\
**Post date:** [April 6, 2017, 8:11am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/1 "2017-04-06T08:11:10Z")

</div>

I am getting the following error while trying to load a saved model.

`KeyError: 'unexpected key "module.encoder.embedding.weight" in state_dict'`

This is the function I am using to load a saved model. (as suggested in [http://pytorch.org/docs/notes/serialization.html#recommend-saving-models](http://pytorch.org/docs/notes/serialization.html#recommend-saving-models))

```
def load_model_states(model, tag):
    """Load a previously saved model states."""
    filename = os.path.join(args.save_path, tag)
    with open(filename, 'rb') as f:
        model.load_state_dict(torch.load(f))

```

The model is a sequence-to-sequence network whose init function (constructor) is given below.

```
def __init__ (self, dictionary, embedding_index, max_sent_length, args):
    """"Constructor of the class."""
    super(Sequence2Sequence, self). __init__ ()
    self.dictionary = dictionary
    self.embedding_index = embedding_index
    self.config = args
    self.encoder = Encoder(len(self.dictionary), self.config)
    self.decoder = AttentionDecoder(len(self.dictionary), max_sent_length, self.config)
    self.criterion = nn.NLLLoss() # Negative log-likelihood loss

    # Initializing the weight parameters for the embedding layer in the encoder.
    self.encoder.init_embedding_weights(self.dictionary, self.embedding_index, self.config.emsize)

```

When I print the model (sequence-to-sequence network), I get the following.

```
Sequence2Sequence (
  (encoder): Encoder (
    (drop): Dropout (p = 0.25)
    (embedding): Embedding(43723, 300)
    (rnn): LSTM(300, 300, batch_first=True, dropout=0.25)
  )
  (decoder): AttentionDecoder (
    (embedding): Embedding(43723, 300)
    (attn): Linear (600 -> 12)
    (attn_combine): Linear (600 -> 300)
    (drop): Dropout (p = 0.25)
    (out): Linear (300 -> 43723)
    (rnn): LSTM(300, 300, batch_first=True, dropout=0.25)
  )
  (criterion): NLLLoss (
  )
)

```

So, `module.encoder.embedding` is an embedding layer, and `module.encoder.embedding.weight` represents the associated weight matrix. So, why it says- `unexpected key "module.encoder.embedding.weight" in state_dict`?

---

<div class="post-metadata">

**Author:** ![fmassa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/32/58_2.png) [@fmassa](https://discuss.pytorch.org/u/fmassa)\
**Post date:** [April 6, 2017, 9:07am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/2 "2017-04-06T09:07:55Z")

</div>

You probably saved the model using `nn.DataParallel`, which stores the model in `module`, and now you are trying to load it without `DataParallel`. You can either add a `nn.DataParallel` temporarily in your network for loading purposes, or you can load the weights file, create a new ordered dict without the `module` prefix, and load it back.

---

<div class="post-metadata">

**Author:** ![wasiahmad](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasiahmad/32/248_2.png) [@wasiahmad](https://discuss.pytorch.org/u/wasiahmad)\
**Post date:** [April 6, 2017, 9:14am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/3 "2017-04-06T09:14:53Z")

</div>

Yes, I used `nn.DataParallel`. I didn’t understand your second suggestion. Loading the weights file, create a new ordered dict without the module prefix and load it back. (Can you provide an example?)

Are you suggesting something like this? (example taken from here - [https://github.com/OpenNMT/OpenNMT-py/blob/master/train.py](https://github.com/OpenNMT/OpenNMT-py/blob/master/train.py))

```
model_state_dict = model.module.state_dict() if len(opt.gpus) > 1 else model.state_dict()
model_state_dict = {k: v for k, v in model_state_dict.items() if 'generator' not in k}
generator_state_dict = model.generator.module.state_dict() if len(opt.gpus) > 1 else model.generator.state_dict()
# (4) drop a checkpoint
checkpoint = {
	'model': model_state_dict,
	'generator': generator_state_dict,
	'dicts': dataset['dicts'],
	'opt': opt,
	'epoch': epoch,
	'optim': optim
}
torch.save(checkpoint,
		   '%s_acc_%.2f_ppl_%.2f_e%d.pt' % (opt.save_model, 100*valid_acc, valid_ppl, epoch))  

```

May I ask you one question about the above code snippet, what is `generator` here?

---

<div class="post-metadata">

**Author:** ![fmassa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/32/58_2.png) [@fmassa](https://discuss.pytorch.org/u/fmassa)\
**Post date:** [April 6, 2017, 9:18am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/4 "2017-04-06T09:18:58Z")

</div>

I was thinking about something like the following:

```python
# original saved file with DataParallel
state_dict = torch.load('myfile.pth.tar')
# create new OrderedDict that does not contain `module.`
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove `module.`
    new_state_dict[name] = v
# load params
model.load_state_dict(new_state_dict)

```

---

<div class="post-metadata">

**Author:** ![wasiahmad](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasiahmad/32/248_2.png) [@wasiahmad](https://discuss.pytorch.org/u/wasiahmad)\
**Post date:** [April 6, 2017, 9:26am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/5 "2017-04-06T09:26:04Z")

</div>

First of all, thanks a lot, by adding a `nn.DataParallel` temporarily in my network for loading purposes worked. Even I tried your second suggested approach, it worked for me as well. Thanks a lottt 🙂

---

<div class="post-metadata">

**Author:** ![achaiah](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/achaiah/32/3366_2.png) [@achaiah](https://discuss.pytorch.org/u/achaiah)\
**Post date:** [April 26, 2017, 10:07pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/7 "2017-04-26T22:07:17Z")

</div>

@wasiahmad By adding `nn.DataParallel` temporarily into your network did you have to have the same number of GPUs available to load the model as when you saved the model?

---

<div class="post-metadata">

**Author:** ![Tingfan\_Wu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/e9bcb4/32.png) [@Tingfan\_Wu](https://discuss.pytorch.org/u/Tingfan_Wu)\
**Post date:** [July 11, 2017, 11:53pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/8 "2017-07-11T23:53:38Z")

</div>

A related question, given the fact we see that saving DataParallel wrapped model can cause problems when the model\_state\_dict is loaded into an unwrapped model. Would one recommend to save the “unwrapped” ‘module’ field inside a DataParallel instance instead ?

> <https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/data_parallel.py#L49>

---

<div class="post-metadata">

**Author:** ![Eugenio\_Culurciello](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eugenio_culurciello/32/52_2.png) [@Eugenio\_Culurciello](https://discuss.pytorch.org/u/Eugenio_Culurciello)\
**Post date:** [July 29, 2017, 3:34pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/9 "2017-07-29T15:34:58Z")

</div>

here is our way for alexnet trained with pytorch examples imagenet:

> [@On a cpu device, how to load checkpoint saved on gpu device](https://discuss.pytorch.org/t/on-a-cpu-device-how-to-load-checkpoint-saved-on-gpu-device/349/15):
>
> We have trained an Alexnet with pytorch examples imagenet ([https://github.com/pytorch/examples/blob/master/imagenet/main.py](https://github.com/pytorch/examples/blob/master/imagenet/main.py)) and have been struggling to convert the model for use on CPU and for inference only. Here is a solution for AlexNet: [https://github.com/e-lab/pytorch-toolbox/blob/master/convert-save-load.md](https://github.com/e-lab/pytorch-toolbox/blob/master/convert-save-load.md) It would be nice to have something more generic…

---

<div class="post-metadata">

**Author:** ![Daniel\_Du](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/8e8cbc/32.png) [@Daniel\_Du](https://discuss.pytorch.org/u/Daniel_Du)\
**Post date:** [December 13, 2017, 9:57pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/11 "2017-12-13T21:57:16Z")

</div>

This works for me. Thanks a lot !

---

<div class="post-metadata">

**Author:** ![datdddd](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b3f665/32.png) [@datdddd](https://discuss.pytorch.org/u/datdddd)\
**Post date:** [March 14, 2018, 8:38am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/12 "2018-03-14T08:38:10Z")

</div>

I am having the same problem, and using the trick with OrderedDict does not work. I am using pytorch 0.3 in the case anything has changed.

I have an word embedding layer that was trained along with the classification task. Training was successful, but loading the model gave the error

`Traceback (most recent call last): File "source/test.py", line 72, in <module> helper.load_model_states_from_checkpoint(model, args.save_path + 'model_best.pth.tar', 'state_dict', args.cuda) File "/u/flashscratch/flashscratch1/d/datduong/universalSentenceEncoder/source/helper.py", line 55, in load_model_states_from_checkpoint model.load_state_dict(checkpoint[tag]) File "/u/home/d/datduong/project/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 490, in load_state_dict .format(name)) KeyError: 'unexpected key "embedding.embedding.embedding.weight" in state_dict'`

The key embedding.embedding.embedding.weight exists (see image). Please let me know what to do.

 ![Capture2](https://discuss.pytorch.org/uploads/default/original/2X/6/6453bb757041e5dc6d09ba143a3822d90bbe3455.PNG)

---

<div class="post-metadata">

**Author:** ![Oktai15](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oktai15/32/5942_2.png) [@Oktai15](https://discuss.pytorch.org/u/Oktai15)\
**Post date:** [April 11, 2018, 1:46pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/13 "2018-04-11T13:46:01Z")

</div>

> [@fmassa](#):
>
> # original saved file with DataParallel
> 
> state\_dict = torch.load(‘myfile.pth.tar’)
> 
> # create new OrderedDict that does not contain `module.`
> 
> from collections import OrderedDict  
> new\_state\_dict = OrderedDict()  
> for k, v in state\_dict.items():  
> name = k[7:] # remove `module.`  
> new\_state\_dict[name] = v
> 
> # load params
> 
> model.load\_state\_dict(new\_state\_dict)

In my opinion, this question-answer should be in something FAQ 🙂

---

<div class="post-metadata">

**Author:** ![Wyn\_Mew](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wyn_mew/32/2313_2.png) [@Wyn\_Mew](https://discuss.pytorch.org/u/Wyn_Mew)\
**Post date:** [April 23, 2018, 6:44am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/14 "2018-04-23T06:44:12Z")

</div>

Check out your saved model file:

```python
check_point = torch.load('myfile.pth.tar')
check_point.key()

```

You may find out your ‘check\_point’ got several keys such as ‘state\_dict’ etc.

```python
checkpoint = torch.load(resume)
state_dict =checkpoint['state_dict']

from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove 'module.' of dataparallel
    new_state_dict[name]=v

model.load_state_dict(new_state_dict)

```

---

<div class="post-metadata">

**Author:** ![dashesy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dashesy/32/4869_2.png) [@dashesy](https://discuss.pytorch.org/u/dashesy)\
**Post date:** [June 12, 2018, 3:07am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/15 "2018-06-12T03:07:43Z")

</div>

What about `nn.DistributedDataParallel`, it seems `DistributedDataParallel` and `DataParallel` can load each other’s parameters.  
Is there an official way to save/load among `DDP/DP/None`?

---

<div class="post-metadata">

**Author:** ![17764591637](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/1/34f0e0/32.png) [@17764591637](https://discuss.pytorch.org/u/17764591637)\
**Post date:** [June 29, 2018, 6:14am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/16 "2018-06-29T06:14:07Z")

</div>

just do this:

model = torch.load(train\_model)  
…  
net.load\_state\_dict(model[‘state\_dict’])

it works for me!

---

<div class="post-metadata">

**Author:** ![harryfyodor](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/e19b73/32.png) [@harryfyodor](https://discuss.pytorch.org/u/harryfyodor)\
**Post date:** [July 17, 2018, 7:36am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/17 "2018-07-17T07:36:41Z")

</div>

Thanks a lot, this worked for me.

---

<div class="post-metadata">

**Author:** ![Alon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alon/32/5661_2.png) [@Alon](https://discuss.pytorch.org/u/Alon)\
**Post date:** [July 23, 2018, 2:09pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/19 "2018-07-23T14:09:15Z")

</div>

Instead of deleting the “module.” string from all the state\_dict keys, you can save your model with:  
`torch.save(model.module.state_dict(), path_to_file)`  
instead of  
`torch.save(model.state_dict(), path_to_file)`  
that way you don’t get the “module.” string to begin with…

---

<div class="post-metadata">

**Author:** ![stoneyang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stoneyang/32/5962_2.png) [@stoneyang](https://discuss.pytorch.org/u/stoneyang)\
**Post date:** [August 3, 2018, 1:51pm UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/20 "2018-08-03T13:51:10Z")

</div>

Thanks for your hints! It saved [my time](https://discuss.pytorch.org/t/missing-keys-unexpected-keys-in-state-dict-when-loading-self-trained-model/22379/2)🌹

---

<div class="post-metadata">

**Author:** ![amy\_hu](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c0e974/32.png) [@amy\_hu](https://discuss.pytorch.org/u/amy_hu)\
**Post date:** [November 12, 2018, 8:05am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/21 "2018-11-12T08:05:56Z")

</div>

that’s work simple and perfect for me! thanks

---

<div class="post-metadata">

**Author:** ![KaiyangZhou](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaiyangzhou/32/3850_2.png) [@KaiyangZhou](https://discuss.pytorch.org/u/KaiyangZhou)\
**Post date:** [February 15, 2019, 11:09am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/22 "2019-02-15T11:09:12Z")

</div>

In case someone needs, this [function](https://github.com/KaiyangZhou/deep-person-reid/blob/master/torchreid/utils/torchtools.py#L104) can handle loading weights w/ and w/o ‘module’.

To save model without ‘module’, you may try [this](https://github.com/KaiyangZhou/deep-person-reid/blob/master/torchreid/utils/iotools.py#L41).

---

<div class="post-metadata">

**Author:** ![weblucas](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/b487fb/32.png) [@weblucas](https://discuss.pytorch.org/u/weblucas)\
**Post date:** [March 15, 2019, 7:40am UTC](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/23 "2019-03-15T07:40:43Z")

</div>

After pytorch 1.xx  
this was fixed, now you only need to do this

```auto
            if isinstance(args.pretrained, torch.nn.DataParallel):
                args.pretrained = args.pretrained.module

```

[Next page](https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686.md?page=2)