Hi @ptrblck, I came across the same issue and was intrigued by your remark:
For one, isn’t it good to be able to process/load the data using a separate process, apart from the training process(es), in order to not tax them with extra processing load? Further, although it’s not my use case, loading the data might involve some heavy preprocessing, warranting even more than one data loader workers per training process; e.g. in the case in of some vision applications.
Maybe you have a completely different idea about this and give us your insight in to this topic.
Thanks in advance.