How to parallelize if my network can not batch?

[dragen (Jackie)]

In my network, it’s highly dynamic so that every data will run different path in the network. In this case, I can not do batch because the data in same batch do not have same work pipeline. But I want to speedup my training, which means, I need some kind of parallism to accelerate the training in the case that I can only train one data per forward procedure.
Any One have some suggestions?

[jpeg729]

There is pytorch.multiprocessing but I haven’t taken the time to understand how it works yet. If you find yourself constrained to giving batches to your model, you could use batches of size 1, and alter your model to squeeze the batch dimension when necessary.

Personally I would try using joblib as in the following example. Again, I have not yet tried it, but I think it would work.

Hyperparameter Optimisation - Multiple Models Multiple Cores

I saw that one too, but it didn’t seem to fit my use case either. My approach would be something like this… from joblib import Parallel, delayed results = Parallel(n_jobs=-1)([delayed(train_function)(args) for _ in range(80)] Where train_function trains a model for a fixed number of epochs or until some stopping criterion and returns a list of validation losses per epoch (for example). When the parallel jobs are all done results is simply a list containing the return values from each run of tr…

[kroscoo (Kris Cao)]

DyNet does autobatching for you, but it’s not Pytorch.

[dragen (Jackie)]

Hi, But my network can not batch since the work pipeline of the sample in same batch is diverse. I suppose Dynet can not do it.

[dragen (Jackie)]

Hi, yes, currently I use [1, c, h, w] to train my network, it’s working but slow. I want to get sort of parallism method.

[Chaofei (Chaofei Hong)]

I have the same problem with you here. have you got a better solution?