How to make a GPU dedicated to data processing

[Sangyeon_Kim]

Hi, I’m working on building a model via weakly-supervised learning.

In order to achieve data generation in a weakly-supervised manner during training process, I’d like to make a GPU dedicated to data processing and pass them to other GPUs, while other GPUs learn using the processed data in parallel.

Is it possible? and could you give me some example or snippet?
I’m thinking of it is important for data synchronization among a data-dedicated GPU and other GPUs.

And I’m also curious it’s possible to select GPU’s device in data loader.

[gcramer23 (Garrett Cramer)]

Hello Sangyeon_Kim,

This might help you with your implementation.

@Sangyeon_Kim

I’d like to make a GPU dedicated to data processing

You can use multiple processes and assign a process to be the data processer. The data processer would use a GPU device.

@Sangyeon_Kim

pass them to other GPUs

You can communicate the processed data over RPC.

@Sangyeon_Kim

while other GPUs learn using the processed data in parallel.

Each trainer will be a process. GPU devices will be assigned to the process. Each trainer will create a DDP instance.

ddp doc Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation
distributed rpc doc Distributed RPC Framework — PyTorch 2.1 documentation

[Sangyeon_Kim]

Thanks gcramer23

I’ll give it a shot.