Why does the shuffle operation take a lot of time?

1

I have millions of 16x16 pixels images and when I set shuffle=False, the training is fast.
But…When I set shuffle=True, so slow…
How to speed up?

2

It’s a matter of how do you save images rather than dataloader itself.
Shuffling just shuffles a list of indices. It seems your system is slower accessing non-sequential lists of indices.
How do you do so?

3

Hi,

Do you mean slow in runtime? Or slow to learn?

4

Thanks for you reply. I mean slow in run time.

5

Thanks for your reply, I have solved the problem. Thank you

6

@QuanHei Hi, same problem here, how do you solve the problem?
I have a very large dataset(1200000 images) and training with shuffle=False is fast while training with shuffle=True is slower and slower.

Any suggesstion or help would be appreciated, thanks!