I just load the whole dataset in the init() of DataLoader, others are just as normal. Be sure to make num_worker < physical cpu . I don’t know why, but it works well when your memory is limited. E.g., my dataset is about 30 GB, I can run on a two-GPU machine with 80 GB memory. Num_worker and physical cpu are 10 and 12 respectively.