Parallelizing a training code using a couple of P100/V100 GPUs

Mona_Jalal · October 8, 2020, 5:08am

Is there a way to parallelize the code in this repo which is written in PyTorch? The training approximately takes 1 month on a 1080Ti GPU, so I am wondering if I can run it much faster say on 8 P100 or V100 GPUs?
I am not familiar which part of code I should change to do so?

I am running it with this command:
$ python train.py --cnn_weight [YOUR HOME DIRECTORY]/.torch/resnet152-b121ed2d.pth

I am using PyTorch 1.6.0

Their train.py code is here:

github.com

furkanbiten/GoodNews/blob/master/train.py

# Use tensorboard

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.optim as optim
import time
from six.moves import cPickle
import opts
import models
from dataloader import *
import eval_utils
import misc.utils as utils
try:
    import tensorflow as tf

This file has been truncated. show original

ptrblck · October 10, 2020, 10:34am

You could use a data parallel approach to use multiple GPUs for the training.
This tutorial gives you an overview of different approaches.