Pipeline batches during inference

[adizhol]

Hi,
Google published a paper where they “pipeline” the inference (and backprop) stage (each later at a time).

Sideways: Depth-Parallel Training of Video Models

I thought that most frameworks do this during the forward pass when inferencing, since you don’t need to wait for the batch to finish calculating all the outputs before loading a new batch.
If that’s not the case, is any work on this planned for pytorch?

Thanks!

[marksaroufim]

@adizhol

I thought that most frameworks do this during the forward pass when inferencing, since you don’t need to wait for the batch to finish calculating all the outputs before loading a new batch.

It’s here Pipeline Parallelism — PyTorch 2.1 documentation