Hello. I would like to perform train/val/test split on my time series dataset. The data is stored in a MariaDB database, thus I had to create custom Pytorch dataset, that actually comprises of TWO Dataset classes:
First dataset is responsible for getting chunk of indices of database rows to be read, so that we will load data in chunks to diminish the memory usage.
Example of generating chunk indices:
DB Table
table chunks
----- ------
1. 1.
2. 2.
3. --> 3.
4. ------
5. 3.
4.
5.
The second dataset will then create sliding windows within these chunks, and fetch corresponding data from databse.
Example of generating sliding windows within chunks of data:
DB Table Sliding
table chunks windows
----- ------ ------
1. 1. 1., 2.
2. 2. --> ------
3. --> 3. 2., 3.
4. ------ ------
5. 3. 3., 4.
4. --> ------
5. 4., 5.
How to perform train/val/test spliting of such dataset. I heard about torch.utils.data.random_split(dataset, lengths) but i don’t want my data to be split randomly. Is there any tool that can be used with Time series and Pytorch Dataset, so that I don’t have to create sepparate Dataset classes for each train/val/test set?
Any hint will be appreciated.