I want to share some NamedTuple (such as mytuple) between different rank. The following codes is possible?
tcpstore = dist.TCPStore(MASTER_ADDR, MASTER_PORT, world_size, MASTER_ADDR == LOCAL_ADDR) dist.init_process_group('nccl', store= tcpstore, rank=rank, world_size=world_size) if rank == 0: store.set("my1", mytuple) else: id = store.get("my1",mytuple)
I have read the
How to store embeddings from different ranks in DistributedDataParallel mode? - #4 by mrshenli . But I want to know if I have 8GPU, how could I init and pass 8 simplequeue?