Feeding in a npy file

1

Hi, I’m completely new to PyTorch, only having experiences with Tensorflow.

I have datasets with shape [?, 128, 2048], with ? being the data number (This number can change).
The image size is [128, 2048], and I have saved them by shape [500, 128, 2048]
I want to feed these data into the neural network using PyTorch.

I heard that there is a common way in PyTorch that is used by most, and I need some help with it…
I think that putting thousands of [128, 2048] on a RAM at once will trigger a memory error, so I want a way to avoid this problem.
Thanks in advance.

2

Hi P_C,

If I understand correctly, you have several .npy files saved, each with five hundred 128x2048 arrays, and would like to use all of these .npy files in a dataset? Have you tried something akin to this link?

Best,
Adam