Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 3. Your PyTorch installation may be too old

1

I trained a model with Colab TPU using torch XLA, every time I load the saved weight I got this error.

My pytorch version is 1.5.0.

I can’t even install the XLA in my machine. And can’t find any way to install the 1.6.

How to solve this and what’s the reason behind this feature?

2

Interestingly, I can just open the file with Xarchiver and change the version to 3 and everything works.

Makes me really curious about the reason behind this.

3

Hi,

It happens because when we make breaking changes in the serialization format, we increase the archive version to make sure that the version of pytorch loading the file knows how to handle all the content of the file.

The “right” fix is to upgrade your version of pytorch to at least the version that was used to create the saved file.

4

Which means we need a backward compatible save. Or do we already have one? I can’t find any after several hours of searching (backward is a method name after all -_-).

Because most of the time the save function only write weight, and almost all of them are in either float16 or float32 which aren’t suppose to be any problem no matter what version.

5

We do have a backward compatible save.
What you try to do here is forward compatible: an old version of pytorch is able to load a file created by a newer version of pytorch. This is significantly harder to do than backward compatibility and so we don’t guarantee it.

6

How to do it?

Honestly all I can find is about nn.Module.backward().

7

The default save is backward compatible.
Meaning that a model saved with an old version can always be loaded with a newer version.

8

No, I won’t be here otherwise. At least not with the 1.6

9

The error you mention is about forward compatibility: " Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 3."
The file’s version comes from a newer version of pytorch than the version you try to load the file with.

10

What’s with it?
Yes, and I’m asking how to use save it with backward compatible, save a single float from 1.6 which is not without bugs to have a 1.5 load it.

Which point you don’t understand.

It make no sense if an error can be fixed by having user edit one single character in weight file and absolutely everything works.

11

Or you want me to start a new thread?

12

save a single float from 1.6 which is not without bugs to have a 1.5 load it.

This is forward compatibility. Not backward compatibility !

It make no sense if an error can be fixed by having user edit one single character in weight file and absolutely everything works .

This happens for most library that use version for their serialization format. The library can only load the a given set of formats. It won’t be able to load the future ones.

Is there a specific reason why you want to be able to do this?

13

The first line of the saved files from torch 1.6.0a0+541814f ends with this

versionFB\x12\x00ZZZZZZZZZZZZZZZZZZ4

Which’s not gonna be opened in the latest torch in pip or conda even though it just ResNet weights.

But!!!
If you change it to

versionFB\x12\x00ZZZZZZZZZZZZZZZZZZ3

Now everything works.

My question all along is how to make 1.6 save weight with x00ZZZZZZZZZZZZZZZZZZ3?
How could this possibly be forward?

After all, all you need to do to make the saved weight backward compatible is just change a letter.

14

For reasons, it just I accidentally use torch-xla nightly in a TPU machine and that nightly come along with torch 1.6.

Training ResNet50 with 20 times faster than my GPU, which is great. But then I can’t load the saved weights to make use of it even though I’m using the 1.5.0.

The latest pip and conda I found is also 1.5, I’m not gonna try upgrade my PC lib to compile 1.6 it in risk of breaking something else, happens a lot as you might expect from Ubuntu, thus this situation.

If you want to say “just update it and everything will work”, even though I only need to change one character and it’ll work too.

Question is, what cause more trouble between

  1. Pytorch loading the file has a chance of doesn’t know how to handle some content of the file, even though most of the time nothing’s gonna happen
  2. Raise the error if version mismatch regardless of what’s in the file, even though most Pytorch users never use save() and load() for anything other than some float32
15

How could this possibly be forward ?

“save with 1.5” → “load with 1.6” is backward compat: files created with older version can still be loaded with newer versions.
“save with 1.6” → “load with 1.5” is forward compat: the older version needs to know how to load newer files.

After all, all you need to do to make the saved weight backward compatible is just change a letter.

Yes for your particular case it just works. But depending on what you save, it might fail unpredictably.
Since 99% of people will never need forward compatibility, we did not added the complexity in the code to make this work, maintain it and test it.

The latest pip and conda I found is also 1.5,

You can select the nightly build on the get started page: Start Locally | PyTorch to get a more recent version without having to compile yourself. These are the same ones that xla uses IIRC.

16

how to use xarchiver on Mac?

17

I think other zip readers should work too though.

On my case, when I opened the saved file with text editor (actually Pycharm) the first line looks like this

PK                    e a checkpoint-032epoch/versionFB ZZZ4

I then just change the last letter from 4 to 3, like

a = open('w1.pth', 'rb').readlines()
a[0] = a[0][:-2] + bytes(b'3\n')
with open('w2.pth', 'wb') as wr:
    for aa in a:
        wr.write(aa)

Don’t forget to backup your file though.

1 Like