Hello, I am experimenting with using multiple GPUs on my university cluster, but I do not see any speed increase when doing so. I am curious why this is. Below I share some data and code.
Duration of 3 epochs’ worth of training:
Using 1 Tesla V100-SXM2-32GB:
- 6 minutes 1 second
- 5 minutes 55 seconds
Using 2 Tesla V100-SXM2-32GB:
- 6 minutes 4 seconds
- 5 minutes 39 seconds
Using 4 Tesla V100-SXM2-32GB:
- 6 minutes 11 seconds
- 5 minutes 46 seconds
Relevant code segments:
Checking GPU status:
print("Is a GPU available?", torch.cuda.is_available())
print("How many GPUs are available?", torch.cuda.device_count())
print("What's the current GPU number?", torch.cuda.current_device())
print("Where's the first GPU?", torch.cuda.device(0))
print("What are the names of the GPUs?")
for i in range(torch.cuda.device_count()):
print(torch.cuda.get_device_name(i))
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print('This should say cuda if the GPUs are set up properly:', device)
Output for the above (in the 4 GPU job):
Is a GPU available? True
How many GPUs are available? 4
What's the current GPU number? 0
Where's the first GPU? <torch.cuda.device object at 0x2b44157750a0>
What are the names of the GPUs?
Tesla V100-SXM2-32GB
Tesla V100-SXM2-32GB
Tesla V100-SXM2-32GB
Tesla V100-SXM2-32GB
This should say cuda if the GPUs are set up properly: cuda
Batch size:
batch_size = 64
Model training function (mostly copied from a PyTorch tutorial):
def train_model(model, dataloaders, criterion, optimizer, num_epochs=25, is_inception=False):
since = time.time()
val_acc_history = []
train_acc_history = []
best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0
for epoch in range(num_epochs):
print('Epoch {}/{}'.format(epoch, num_epochs - 1))
print('-' * 10)
# Each epoch has a training and validation phase
for phase in ['train', 'val']:
if phase == 'train':
model.train() # Set model to training mode
else:
model.eval() # Set model to evaluate mode
running_loss = 0.0
running_corrects = 0
# Iterate over data.
for inputs, labels in dataloaders[phase]:
inputs = inputs.to(device)
labels = labels.to(device)
# zero the parameter gradients
optimizer.zero_grad()
# forward
# track history if only in train
with torch.set_grad_enabled(phase == 'train'):
# Get model outputs and calculate loss
# Special case for inception because in training it has an
# auxiliary output. In train mode we calculate the loss by
# summing the final output and the auxiliary output but in
# testing we only consider the final output.
if is_inception and phase == 'train':
# From https://discuss.pytorch.org/t/how-to-optimize-inception-model-with-auxiliary-classifiers/7958
outputs, aux_outputs = model(inputs)
loss1 = criterion(outputs, labels)
loss2 = criterion(aux_outputs, labels)
loss = loss1 + 0.4*loss2
else:
outputs = model(inputs)
loss = criterion(outputs, labels)
_, preds = torch.max(outputs, 1)
# backward + optimize only if in training phase
if phase == 'train':
loss.backward()
optimizer.step()
# statistics
running_loss += loss.item() * inputs.size(0)
running_corrects += torch.sum(preds == labels.data)
epoch_loss = running_loss / len(dataloaders[phase].dataset)
epoch_acc = running_corrects.double() / len(dataloaders[phase].dataset)
print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))
# deep copy the model
if phase == 'val' and epoch_acc > best_acc:
best_acc = epoch_acc
best_model_wts = copy.deepcopy(model.state_dict())
if phase == 'val':
val_acc_history.append(epoch_acc)
if phase == 'train':
train_acc_history.append(epoch_acc)
print()
torch.save({
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
# 'best_model_accuracy': best_acc,
# 'best_model_weights': best_model_wts,
'loss': loss,
# 'train_hist': train_acc_history,
# 'val_hist': val_acc_history,
# }, f'/project/rrg-lelliott/jsa378/model_1_output/checkpoint_run_{run}_epoch_{epoch}.tar')
}, str(os.getenv('location3'))+f'/checkpoint_run_{run}_epoch_{epoch}.tar')
torch.save({
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'best_model_accuracy': best_acc,
'best_model_weights': best_model_wts,
'loss': loss,
'train_hist': train_acc_history,
'val_hist': val_acc_history,
# }, f'/project/rrg-lelliott/jsa378/model_1_output/checkpoint_run_{run}.tar')
}, str(os.getenv('location3'))+f'/checkpoint_run_{run}.tar')
time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))
# load best model weights
model.load_state_dict(best_model_wts)
return model, val_acc_history, train_acc_history
Model initialization and placement on GPU:
model_ft, input_size = initialize_model(model_name, num_classes, feature_extract, use_pretrained=True)
print(model_ft)
if torch.cuda.device_count() > 1:
print("Let's use", torch.cuda.device_count(), "GPUs!")
model_ft = nn.DataParallel(model_ft)
model_ft = model_ft.to(device)
Questions and Comments:
Below I’ve listed a few questions that come to mind.
- Are there any apparent errors in my code (for the purpose of using multiple GPUs)?
- Do I need to double or quadruple my batch size to see an increase in training speed?
- Is it possible that the cost of inter-GPU communication roughly cancels out any speed increase?
- My data set is only ~14,000 images, about 85 MB total. I am using the off-the-shelf AlexNet. Is it possible that my model and data are too small to benefit from multiple GPUs?
I’m interested in using multiple GPUs because my advisor and I are toying with the idea of randomly splitting the data into training and test sets 100 times, and training for 200 epochs each. 20,000 epochs at 2 minutes per epoch is like 4 weeks, hence my interest in speeding things up.
I appreciate any help with this.