How to parallelize if my network can not batch?

In my network, it’s highly dynamic so that every data will run different path in the network. In this case, I can not do batch because the data in same batch do not have same work pipeline. But I want to speedup my training, which means, I need some kind of parallism to accelerate the training in the case that I can only train one data per forward procedure.
Any One have some suggestions?

1 Like

There is pytorch.multiprocessing but I haven’t taken the time to understand how it works yet. If you find yourself constrained to giving batches to your model, you could use batches of size 1, and alter your model to squeeze the batch dimension when necessary.

Personally I would try using joblib as in the following example. Again, I have not yet tried it, but I think it would work.

DyNet does autobatching for you, but it’s not Pytorch.

Hi, But my network can not batch since the work pipeline of the sample in same batch is diverse. I suppose Dynet can not do it.

Hi, yes, currently I use [1, c, h, w] to train my network, it’s working but slow. I want to get sort of parallism method.

I have the same problem with you here. have you got a better solution?