C++ Multiple GPUs for inference

C++
1

Hi Everyone,

I am unable to find any documentation on how to set multiple GPUs for inference.

In python the following can be done:

device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)

model = CreateModel()

model= nn.DataParallel(model)
model.to(device)

However for C++ I can’t find the equivalent or any documentation.
torch::nn does not contain either DataParallel or DistributedDataParallel

Any help is appreciated, thanks!

2

libtorch uses the parallel::data_parallel API as seen e.g. here. I don’t know if there are other wrappers, but based on the examples I would guess they should work in a similar way as the DataParallel wrapper in Python.

3

Thank you for the snippet!!

I forgot to include

#include <torch/nn.h>
#include <torch/nn/parallel/data_parallel.h>

4

Hi, Sorry to disturb you , I tried the wrapper as you suggest,but the function “parallel::data_parallel” calls another function named “_parallel_run”, which can not be parsed by the compiler of vs2017. It seems like there is no implementation of this function.
Is there any suggestions?

5

Unfortunately, I don’t know what the current status and support of distributed runs on Windows is so can’t help much :confused: Let’s wait for some Windows experts in case they are familiar with it.

6

Thanks for your reply ~