I have this weird bug where I have to load many numpy arrays from disk and each array is 1 .npy file on disk, but when I increase my batch size after a certain point i.e. 64 (this means 64 files should be loaded) I get this cryptic error:
IOError: bad message length. My dataloader is very standard has 8 workers so I’m not sure what the actual issue is.