Each csv is one training example

1

Hi,
I have many csv files that has multiple rows and columns which are mostly floating point numbers (some are categorical but one-hot encoded).
Each csv file is the representation of one training example. It contains dependent and independent variables in the same file.

Capture
Capture948×472 17.6 KB

** The above image is the representation of one csv

I want to build a simple ann or any other neural net model. I have problem in processing input data. As each csv is one single training example, in which format should i have to store data to pass to a neural net.

Thanks in advance,
skw

2

Hi,

You can build a custom data loader, something like this. Where you initialize your data loaders with all path of all CSV files and in __getitem__ method you can load CSV and return input and target. If your data is small enough to easily fit into your memory then you can load all CSV files in data loader initialization itself.

3

Hi,
Thank you for the reply.
I forgot to mention that, the length/shape of the csv (only rows) may vary