If you’ve two models net1 and net2 sitting on different GPUs and two optimizers optim1 and optim2 taking care of net1 and net2 respectively - how would you execute optim1.step() and optim2.step() in parallel? any help/pointers to tutorials to do this are appreciated! thanks!
CUDA operations are run asynchronously, so just
optim1.step()
optim2.step()