Using "torch.utils.data.Dataset" and "torch.utils.data.DataLoader"

1

Hello everyone.

I know this has been asked several times but I still have some doubts.

I am training a NN to fit a 3D function. Then, I have a file with four columns, three of them are the input values and the forth is the value of the function at the point. Say I train the NN with 1000 - 2000 points, to give you an idea.

The way I am loading data is simply:
data = np.loadtxt(‘path_to_file’)
x = torch.from_numpy(data[:,0:3])
y = torch.from_numpy(data[:,3])

and with that I train the NN.

Checking this forum, I have seen there are options like “torch.utils.data.Dataset” or “torch.utils.data.DataLoader” that seem to be build for this purpose.

Could anyone explain me:
Are they build to load datasets as the ones I am using?
What would be the advantage of using them in my case?

Thank you very much!!

2

They are used when 1) data is too large to load ll at once into memory, and/or 2) needs preprocessing that is most efficiently done in a separate process in order to not block training.