How do I share data between GPUs that is not a tensor?

1

The torch.distributed library seems to only support sharing tensors between GPUs. I want to share arbitrary data between GPUs besides just tensors.

What is the elegant way of doing this?

Since all GPUs are on the same node, can they simply access the same data in CPU memory? If I have a FLAG variable in CPU memory and rank==0 modifies it during runtime, can rank==2 access the modification?

2

I’m not sure which data you would like to share as I would assume the GPU(s) should load it in the end (so a tensor should work).

No, GPUs cannot work on CPU data directly and you would have to move it to the device first.

3

I want to share arbitrary data, not tensors. The data will not be loaded by the GPU. For example, I want to send a dictionary from rank 0, to all devices, and the dictionary will contain only hyperparameters for training.
Is there a way to do this?