Pytorch Forecasting: Loading a custom dataset

Greetings, everyone!

I’m having trouble with loading custom datasets into PyTorch Forecasting. I already posted the question to Stack Overflow but it seems that I might find the answer here here’s the message pasted for your convenience:

I’m trying to load a custom dataset to PyTorch Forecasting by modifying the example given in this Github repository. However I’m stuck at instantiating the TimeSeriesDataSet. The relevant parts of the code are as follows:

import numpy as np
import pandas as pd

df = pd.read_csv("data.csv")
print(df.shape) # (300, 8)

# Divide the timestamps so that they are incremented by one each row.
df["unix"] = df["unix"].apply(lambda n: int(n / 86400))

# Set "unix" as the index
#df = df.set_index("unix")

# Add *integer* indices.
df["index"] = np.arange(300)
df = df.set_index("index")

# Add group column.
df["group"] = np.repeat(np.arange(30), 10)

from pytorch_forecasting import TimeSeriesDataSet

target = ["foo", "bar", "baz"]

# Create the dataset from the pandas dataframe
dataset = TimeSeriesDataSet(
    df,
    group_ids                  = ["group"],
    target                     = target,
    time_idx                   = "unix",
    min_encoder_length         = 50,
    max_encoder_length         = 50,
    min_prediction_length      = 20,
    max_prediction_length      = 20,
    time_varying_unknown_reals = target,
    allow_missing_timesteps    = True
)

And the error message plus traceback:

/home/user/.virtualenvs/torch/lib/python3.9/site-packages/pytorch_forecasting/data/timeseries.py:1241: UserWarning: Min encoder length and/or min_prediction_idx and/or min prediction length and/or lags are too large for 30 series/groups which therefore are not present in the dataset index. This means no predictions can be made for those series. First 10 removed groups: [{'__group_id__group': 0}, {'__group_id__group': 1}, {'__group_id__group': 2}, {'__group_id__group': 3}, {'__group_id__group': 4}, {'__group_id__group': 5}, {'__group_id__group': 6}, {'__group_id__group': 7}, {'__group_id__group': 8}, {'__group_id__group': 9}]
  warnings.warn(

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_822/3402560775.py in <module>
      4 
      5 # create the dataset from the pandas dataframe
----> 6 dataset = TimeSeriesDataSet(
      7     df,
      8     group_ids                  = ["group"],

~/.virtualenvs/torch/lib/python3.9/site-packages/pytorch_forecasting/data/timeseries.py in __init__(self, data, time_idx, target, group_ids, weight, max_encoder_length, min_encoder_length, min_prediction_idx, min_prediction_length, max_prediction_length, static_categoricals, static_reals, time_varying_known_categoricals, time_varying_known_reals, time_varying_unknown_categoricals, time_varying_unknown_reals, variable_groups, constant_fill_strategy, allow_missing_timesteps, lags, add_relative_time_idx, add_target_scales, add_encoder_length, target_normalizer, categorical_encoders, scalers, randomize_length, predict_mode)
    437 
    438         # create index
--> 439         self.index = self._construct_index(data, predict_mode=predict_mode)
    440 
    441         # convert to torch tensor for high performance data loading later

~/.virtualenvs/torch/lib/python3.9/site-packages/pytorch_forecasting/data/timeseries.py in _construct_index(self, data, predict_mode)
   1247                 UserWarning,
   1248             )
-> 1249         assert (
   1250             len(df_index) > 0
   1251         ), "filters should not remove entries all entries - check encoder/decoder lengths and lags"

AssertionError: filters should not remove entries all entries - check encoder/decoder lengths and lags

I have tried tweaking the initializer arguments without success. The file timeseries.py can be found in the same Github repository, at pytorch_forecasting/data/timeseries.py, line 1246. Cannot give a third link in a single post, unfortunately.

What happens if you remove the filtering by using 0 for the min. values and a huge number for the max. ones?

That solves the loading issue. Thank you!