Use joblib to train an ensemble of small models on the same GPU in parallel

[ptrblck]

@SunHaozhe

If I set n_jobs=2 to train 10 base models, I would expect each job trains 5 base models and the 2 jobs are run in parallel, so in theory this should result in 1/2 of the training time compared to using n_jobs=1 to train 10 base models, which means a factor of 2 of training speedup.

That’s not necessarily the case as you would need to have free compute resources on your GPU and would also need to use different streams while still providing enough resources for a parallel execution. If a single stream already saturates the compute resources from your GPU, the stream launches will be serialized as they cannot run in parallel.