The standard batching method is to provide multiple inputs to a single network. Does PyTorch have support for batching calls with the same input to multiple networks, provided those networks have the same architecture, just different weights?
If you want to feed the same input (or batch of inputs) to multiple networks, you can create a list of modules and optimizers. Just use these inside the training, like you would use a single network. Something like this should work.
models = list([mode1, model2,...., model10]) optimizers = list([optimizer1, optimizer2,...., optimizer10]) for model in models: model.train() for batch in train_loader: inputs, targets = batch['input'].to(device), batch['target'].to(device) for c in range(classnum): optimizers[c].zero_grad() : : : outputs = models[c](each_input) loss = criterion(outputs, targets) loss.backward() optimizers[c].step() running_loss += loss.item()
This runs each model on the input sequentially. What I’m looking for is a way to run multiple models on the same input in parallel on the GPU (the same way that one model can run on multiple inputs in parallel instead of sequentially).
You can read the data just once (with the
DataLoader). Then, create multiple models. Next, push each one to the desired device via
to('cuda:id') and simply pass the data to each model.
Now, the training happens on different devices, so it should be executed in parallel.
If you want to train a model using multiple devices, set