Writing and Reading to a Dictionary

1

Hi, I am preprocessing my data, and I want to save my data which is a dictionary of tensors, to a file, and then read this file “live” when running the network.

I tried using Json file, didn’t go well:“TypeError: Object of type ‘Tensor’ is not JSON serializable”

The error is clear, so no questions about that.
I saw two options dedicated to pytorch:

  1. Pickle- Have no Idea about this, found very little information about that.
  2. Use the save and load model given by pytorch, is it possible or superslow/non-effective, other warnings?

What’s your opinion? maybe something else?

Thanks,
Barak.

2

You could use PyTorch’s save and load methods. If you want to lazily load the data, you could store each data sample in a separate file and just load this particular file later (if you are not dealing with too many files).
Otherwise you should also be able to use HDF5 to store and load the data. We’ve had recently some topics about this.