Hello,
I have been using FasterRCNN from Torchvision’s models that uses DistributedDataParallel for training. However, I find that while using multiple GPU, the memory consumption is far more than without multiple GPU. Here is my code
kwargs = {}
kwargs['min_size'] = args.min_size
kwargs['max_size'] = args.max_size
model = ModifiedFRCNN(cfg=cfg, custom_anchor=args.custom_anchor,
use_def=args.use_def, cpm=args.cpm,
default_filter=args.default_filter,
soft_nms=args.soft_nms,
upscale_r=args.upscale_r, **kwargs).cuda().eval()
model = restore_network(model)
model_without_ddp = model
dataset = GenData(args.test_dataset,
args.base_path,
dataset_param=None,
train=False)
if args.n_gpu > 1:
init_distributed_mode(args)
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu],
find_unused_parameters=True)
model_without_ddp = model.module
sampler = torch.utils.data.distributed.DistributedSampler(dataset)
batch_sampler = torch.utils.data.BatchSampler(sampler,
args.batch_size,
drop_last=True)
data_loader = torch.utils.data.DataLoader(dataset,
batch_sampler=batch_sampler,
num_workers=args.num_workers,
collate_fn=coco_collate)
metric_logger = MetricLogger(delimiter=" ")
header = 'Valid:'
batch_iterator = metric_logger.log_every(data_loader, 100, header)
else:
model = model.cuda()
data_loader = iter(data.DataLoader(dataset, args.batch_size, shuffle=False,
num_workers=args.num_workers,
collate_fn=coco_collate))
batch_iterator = iter(data_loader)
ModifiedFRCNN is a class that inherits FRCNN to make trivial changes, such as parameter, postprocessing etc.
Case 1 : When n_gpu=1, I am able to use a batch size of upto 8.
Case 2 : When n_gpu=4, I am unable to even use a batch size of 1.
Both the above mentioned cases are on same the GPU, 2080Ti. Can someone please help me understand what causes this? Here is the command I use to launch the script
python -m torch.distributed.launch --nproc_per_node=4 --use_env test.py <other_arguments> --world_size 4 --n_gpu 4
Thank you and Regards,