It would be helpful if you can provide a initialization function for dataset class which will be called when each sub-process (worker) start. In such case we can do some initialization, like creating separate file descriptors for worker process. Because whether file descriptors are inheritable or not depends on OS and environment, it is hard to control it. But having separate objects like file descriptors for each worker in many cases will speedup data loading a lot.
Or you can add dataset as an argument to worker_init_fn function in DataLoader.