How to make a GPU dedicated to data processing

Hi, I’m working on building a model via weakly-supervised learning.

In order to achieve data generation in a weakly-supervised manner during training process, I’d like to make a GPU dedicated to data processing and pass them to other GPUs, while other GPUs learn using the processed data in parallel.

Is it possible? and could you give me some example or snippet?
I’m thinking of it is important for data synchronization among a data-dedicated GPU and other GPUs.

And I’m also curious it’s possible to select GPU’s device in data loader.

Hello Sangyeon_Kim,

This might help you with your implementation.

You can use multiple processes and assign a process to be the data processer. The data processer would use a GPU device.

You can communicate the processed data over RPC.

Each trainer will be a process. GPU devices will be assigned to the process. Each trainer will create a DDP instance.

ddp doc Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.8.1+cu102 documentation
distributed rpc doc Distributed RPC Framework — PyTorch 1.8.1 documentation

Thanks gcramer23 :slight_smile:

I’ll give it a shot.