Special shape dataloader

Hello. I am trying to prepare some data for training and testing.

I am running in a problem as most of the tutorials i’ve seen are built around image datasets but i’ve got some custom data that i want to use for the Neural Network now…


Basically, let’s say i have 5 CSV files (data1,2,3,4,5) with data and 5 CSV files (state1,2,3,4,5) with states.

The network should use the corresponding data file to guess the value that is written in the state file.

The data files contain 2 columns, X and Y (timeseries) of 100 points each.

The state files only contain 1 column and 1 data, (kept or not kept (0 or 1))

The issue i have now

I can succesfully create a dataloader and input tensors that i create after the CSV files.

I’ll get 3 tensors, 1 for DataXvalues, 1 for DataYvalues, 1 for State

I create a dataLoader like this

myDataLoader = torch.utils.data.DataLoader((myXTensor,myYTensor,myStateTensor))

And it seems to work good, but…

But i only have the stuff i need for 1 train or test pass now, and i don’t see how im supposed to fit the 4 other sets i should add next! (Data2,3,4,5 and State2,3,4,5)

When i lookup my DataLoader the data in it is somewhat like this

[0] - Data for X
[1] - Data for Y
[2] - State

Should i just add a lot of tensors and i could itterate doing jumps of 3 with other sets of data (is that a setting in the DataLoader?) (I would do one pass with Input 0,1 Output 2 - Next pass with Input 3,4 Output 5 - The next one with 6,7,8…)?

If there’s more info required i’ll give all the details i can!

Update on this -

What i am planning to do is a List for Timeseries Data and a List of States

So i would itterate over the lists to secure the proper values for each pass (1 full Timeseries X + Y and the Kept/Not Kept state)

Does this seem like a solution that would work?

Also, i would be treating the XY values of the Timeseries like this

All the X (times) first and then underneat all the Y (Values), Does this make sense?