How to parallelize if my network can not batch?

[jpeg729]

There is pytorch.multiprocessing but I haven’t taken the time to understand how it works yet. If you find yourself constrained to giving batches to your model, you could use batches of size 1, and alter your model to squeeze the batch dimension when necessary.

Personally I would try using joblib as in the following example. Again, I have not yet tried it, but I think it would work.

Hyperparameter Optimisation - Multiple Models Multiple Cores

I saw that one too, but it didn’t seem to fit my use case either. My approach would be something like this… from joblib import Parallel, delayed results = Parallel(n_jobs=-1)([delayed(train_function)(args) for _ in range(80)] Where train_function trains a model for a fixed number of epochs or until some stopping criterion and returns a list of validation losses per epoch (for example). When the parallel jobs are all done results is simply a list containing the return values from each run of tr…