Using continuous data as training data

1

Hello folks,

I am back with a question about the data loading process.
I explain myself : I have model I want to train over some data but the data is already downloaded for some of it and the rest is continuously (every minutes) downloaded from a web API.
Would it be possible to load it as it comes and train on it ? Or should I limit myself waiting for a good amount to be downloaded, train on it and then wait for the next chunk to be here ?

Thanks in advance

Justin

2

Both ways you described are possible in PyTorch. I think this is more of a design question than a PyTorch question.

3

I know how to do it with data downloaded, do you have an example for continious data ?
And what would be the most efficient way to do it considering the time of training and time to fetch the data ?