Hello dear Torch firends!
My problem is the following, I have a fairly large dataset that is stored in .tfrecord format. Since I am way to deep into the project to switch to tensorflow I would like to train my model with this additional data using Pytorch. The issue is that am not sure how to parse the binary stream stored in .tfrecord as a pytorch dataset, also the dataset is to large to be completely stored in memory.
I was thinking this problem as well not long ago. I didn’t find a solution, but use a simple workaround which is to parse tfrecords using TF and save them into a different format such as hdf5, which can then be used in Pytorch.
I guess it would be quite difficult to parse tfrecords in Pytorch, but would be happy to be proved wrong if other people know how to do it.
Thanks for the reply!
This is a solution I implemented for now, however this is far less efficient in terms of memory and loading time compared to tfrecord. It seems pytorch needs its own dataformat… torchrecord?