Hi. I want to use multi-gpus but I don’t know how to use it.
I used the DataParallel function because it seems to be the most basic way.
When I use 2 gpus, the images are well assigned to each gpus, but the speed at which the actual code runs is the same as when using one gpu.
Below is the code I used.
# create model print("=> creating model 'resnet18'") model = resnet.resnet18() if torch.cuda.device_count() > 1: print("Using", torch.cuda.device_count(), "GPUs") # DataParallel will divide and allocate batch_size to all available GPUs model = nn.DataParallel(model.cuda(), device_ids=[0,1]) # define loss function (criterion) and optimizer criterion = nn.NLLLoss() softmax = nn.Softmax(1) optimizer = torch.optim.SGD(model.parameters(), learning_rate, momentum=momentum, weight_decay=weight_decay) cudnn.benchmark = True ## train code def train(train_loader, model, criterion, optimizer, epoch): # switch to train mode model.train() for i, (images, target) in enumerate(train_loader): images = images.to(device) target = target.to(device) out = model(images) out_sm = softmax(out) log = torch.log(out_sm) loss = criterion(log, target) # compute gradient and do SGD step optimizer.zero_grad() loss.backward() optimizer.step() return loss
Is there anything I missed?