Hi, is there a direct wat to get TFRecords dataset as Pytorch Dataset?
Now i am using Tensorflow to get the dataset to numpy and to Torch Tensor.
But , this is slow.
Hi, is there a direct wat to get TFRecords dataset as Pytorch Dataset?
Now i am using Tensorflow to get the dataset to numpy and to Torch Tensor.
But , this is slow.
there isn’t a direct path. via numpy is the way to go
If you are using the dataset often, I would suggest extracting it once and saving it in another format, this way you can have fast data reads when training.
I personnaly used LMDB for a larger-than-memory sized dataset, and data loading is fast enough to saturate the GPU(s) at all times (as advised by @smth in this post BTW).