Using dataloader for both train and test sets

1

Hi everyone!

I’m trying to train an LSTM network to compare to baseline models that I have trained in the past. The problem is that I have a test set that is way too big to load into Cuda all at once for calculating my model’s loss.

However, I would like to use this same test set as I would like to keep it consistent with my baseline models. And I don’t really want to remove elements from my train set either for the same reason.

Does anyone have tips on how to do that? I’m trying to iterate through both train and test DataLoaders at once, but no success so far…

2

Usually you wouldn’t load the entire data into the host RAM or the GPU unless you are working with tiny datasets and are willing to use device memory for the latter case.
Check the data loading tutorial to see how to lazily load the samples.