I would like to train a model where it contains 2 sub-modules. I would like to train sub-model 1 in one gpu and sub-model 2 in another gpu. How would i do in pytorch? I tried specifying cuda device separately for each sub-module but it throws an error.
RuntimeError: tensors are on different GPUs
class Net(nn.Module): def __init__(self): super(Net, self).__init__() self.conv1 = nn.Conv2d(1, 10, kernel_size=5) self.conv2 = nn.Conv2d(10, 20, kernel_size=5).cuda(device=1) self.conv2_drop = nn.Dropout2d() self.fc1 = nn.Linear(320, 50) self.fc2 = nn.Linear(50, 10) def forward(self, x): x = F.relu(F.max_pool2d(self.conv1(x), 2)) x = x.cuda(device=1) conv2_in_gpu1 = self.conv2(x) x = x.cuda(device=0) x = F.relu(F.max_pool2d(self.conv2_drop(conv2_in_gpu1), 2)) x = x.view(-1, 320) x = F.relu(self.fc1(x)) x = F.dropout(x, training=self.training) x = self.fc2(x) return F.log_softmax(x, dim=1) model = Net() if args.cuda: model.cuda(device=0)
This is just an example of what i was trying to achieve. I would like the
self.conv2 to be performed in gpu1 and rest in gpu0.