Resume Dataloader

1

Hello,

I have been assigned several hours per day for using GPU; after which I have to interrupt training and resume during the next available slot. I am able to store everything I need for resuming the training except the state of the dataloader.

I am using LSUN bedroom dataset for training a model. I would like to resume Dataloader from where it stopped the previous day. Can anyone suggest how to do this?

1 Like
2

One way is to write your custom DataLoader with seed value (which will be same for every usage).
And log number of times you have iterated over the dataloader before stopping. When you start again, you need to iterate till the last time using log.

this link might be helpful for random seed based data loader