# Testing Distributed PyTorch code on a single GPU

**URL:** <https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833>\
**Category:** distributed\
**Created:** [August 6, 2023, 8:35pm UTC](https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833 "2023-08-06T20:35:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex\_McKinney](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_mckinney/32/61281_2.png) [@Alex\_McKinney](https://discuss.pytorch.org/u/Alex_McKinney)\
**Post date:** [August 6, 2023, 8:35pm UTC](https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833/1 "2023-08-06T20:35:08Z")

</div>

Greetings,

A rather simple question, that may be complex to answer.

Anyone know how to test distributed PyTorch code on a single GPU? I don’t care how slow it executes, just that it is numerically accurate.

I’m writing a no-nonsense blog on different distributed PyTorch methods, but would rather not have to rent a cluster just to write a blog.

With Jax, I could set XLA environment variable `xla_force_host_platform_device_count` to force a CPU to appear like many for testing purposes. Anything like this exist in PyTorch?

Thanks!

---

<div class="post-metadata">

**Author:** ![H-Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h-huang/32/35598_2.png) [@H-Huang](https://discuss.pytorch.org/u/H-Huang)\
**Post date:** [August 8, 2023, 2:56pm UTC](https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833/2 "2023-08-08T14:56:13Z")

</div>

Hi, thanks for the question. This depends on what aspect of distributed PyTorch you would like to test. For example, if you just need to call collectives (e.g. `dist.allreduce`, `dist.broadcast`) or DDP, then you can just use the gloo backend which operates with CPU (`dist.init_process_group(backend="gloo")`.

If you need something specific to GPU, for example testing NCCL collectives or using FSDP. We don’t currently have a way to virtually split a single GPU to act as multiple.

---

<div class="post-metadata">

**Author:** ![Alex\_McKinney](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_mckinney/32/61281_2.png) [@Alex\_McKinney](https://discuss.pytorch.org/u/Alex_McKinney)\
**Post date:** [August 9, 2023, 2:48pm UTC](https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833/3 "2023-08-09T14:48:35Z")

</div>

> This depends on what aspect of distributed PyTorch you would like to test.  
> Sadly, I want to test… all of them! Lofty goal is 3D parallelism, but would settle for 2D.

So, that would include things like FSDP, NCCL collectives, as well as pipelining.

> We don’t currently have a way to virtually split a single GPU to act as multiple.  
> So, the conclusion is I should probably start using my wallet to rent a small cluster 🤔

I saw a few datacentre inference GPUs do have support for splitting GPUs into virtual ones, but nothing on consumer hardware. If PyTorch doesn’t have support, would you know of anything on a consumer GPU that could work?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 9, 2023, 7:57pm UTC](https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833/4 "2023-08-09T19:57:37Z")

</div>

> [@Alex\_McKinney](#):
>
> I saw a few datacentre inference GPUs do have support for splitting GPUs into virtual ones

I assume you are referring to MIG on NVIDIA data center GPUs. If so, note that no communication between MIG slices is supported and only a single process per MIG (without IPC) is supported, so MIG won’t help.

---

<div class="post-metadata">

**Author:** ![Alex\_McKinney](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_mckinney/32/61281_2.png) [@Alex\_McKinney](https://discuss.pytorch.org/u/Alex_McKinney)\
**Post date:** [August 9, 2023, 8:23pm UTC](https://discuss.pytorch.org/t/testing-distributed-pytorch-code-on-a-single-gpu/185833/5 "2023-08-09T20:23:10Z")

</div>

MIG was one, yes. I think `vgu` may also be a candidate (as in, the subcommand `nvidia-smi vgpu` however this is also not on my hardware. I wasn’t aware that MIG didn’t have support for comms between slices though 😕

Seems this is a dead end for now?
