I’m currently working with sequential data. I have seven features and want to predict one target value. Before I will do the actual prediction with a LSTM, I want to be sure that the training and test data are “similar”/ from the same distribution. But there isn’t something like a trend or a regular behaviour because the data depends on the users input.
If I would know such a metric, the loss of the model on the test data could be interpreted more clearly.
Some classic methods like taking the mean and the std or using correlation or euclidean distance actually don’t help here much, because the mean and std don’t provide any measurement about the behaviour of the signal and I actually don’t want to know if the signals are correlated either.
My approach would be:
- define a window size N , like 20 data points
- get some test sequences with length N from the train and test data
- using tsfresh to calculate features on these time series
- y indicates whether the time series belongs to train or test data
- use a Random Forest classifier to predict the y based on the features given by tsfresh
If the RF is able to classify the time series with high accuracy, the train and test data probably differs too much.
Is this a good approach? Are there any other suggestions or methods I can use?
Is there an algorithm which could give me the probability of an unseen time series, given some example time sequences?
Thanks for helping