I am digging into the torch.save and the data flow of the PyTorch just because I am self-interested how this is using python and c++ to store its data.
So, in my short studying (it’s been just a couple of weeks), I am kinda understood what Pickle does to serialize the data. And now I am stuck at understanding the persistent_id(obj) in torch.save.
By merely googling what is “persistent_id” in Pickle (which is not implemented at first in the Pickle.py), I understood that it makes not to be duplicated so that we don’t serialize the same objects.
However, when I look at the code inside of the “persistent_id(obj)” in serialization.py in PyTorch. It does something else. But, I am totally lost to fully understand what the reason is that it exists.
So, to just briefly explain how I understood the method, I will shortly go over the flow of the code.
- Assume that I call torch.save(model.state_dict(), PATH)
- Assume that the object goes through _save(obj)
- In _save(obj), it initializes Pickler itself and goes into Pickle.py
- In Pickle.py, it uses persistent_id made by Serialization.py in PyTorch
- Persistent_id(obj) firstly filters whether it is tensor or not by checking TypedStorage
- if it is true, it makes object to untyped storage because tensors could be various types
- Then, it is mapping the obj’s data pointer to store the address in memory so that it is not directly saving the tensor data to the memory
- In the method, it stores the address and the index will be returned so that PyTorch can find the address later.
- It returns storage type and index, then Pickle.py serialize the returned information.
In my curiosity by looking at it, is it just made for organizing in the checkpoint file?
This is because I guess persistent_id is not necessary for saving since when we can load the models by just using only Pickle (w/o persistent_id).
What does actually "persistent_id(obj) do in PyTorch?
I am sorry if I am questioning in wrong way in this community because this is my second question in this community and I have no where to ask my curiosity.