Yes. 32 nodes × 1 A100 40 GB is a defensible distributed training testbed, but you should frame it explicitly as a communication-heavy, multi-node configuration, rather than as a representative model of modern multi-GPU-per-node training systems.
The key distinction is between “is this a valid experiment?” and “does this represent a typical production cluster?” Your setup is valid for the first, but has limitations for the second.
Why 1 GPU/node matters
With one GPU per node, essentially every GPU collective has to cross the network:
GPU 0 ── network ── GPU 1 ── network ── GPU 2 ...
Whereas a typical multi-GPU node can have something closer to:
GPU ─ NVLink/NVSwitch ─ GPU
│ │
└────── network ────────┘
Consequently, your experiment will place much more pressure on:
So if your goal is specifically to investigate distributed communication, scaling, or networking, this configuration can actually be quite useful.
If your claim is instead something like “our results predict the performance of a modern 8-GPU-per-node training cluster,” then you would need to be much more cautious.
What I would measure
At minimum, establish a communication baseline before running the actual training experiment.
I’d include:
-
NCCL all-reduce bandwidth
-
All-reduce latency
-
All-gather / reduce-scatter performance, if relevant to your algorithm
-
Network throughput between nodes
-
Scaling from 1 → 2 → 4 → 8 → 16 → 32 GPUs
-
Samples/sec or tokens/sec
-
GPU utilization
-
Percentage of iteration spent communicating
-
Scaling efficiency
For example, define scaling efficiency as
[
E(N)=\frac{T_1}{N T_N},
]
where (T_1) is the single-GPU throughput and (T_N) is the throughput using (N) GPUs.
That gives you a much stronger result than simply reporting that “32 GPUs were faster.”
Particularly important: separate computation from communication
Suppose your iteration takes:
[
T_{\text{iteration}}
T_{\text{compute}}
+
T_{\text{communication}}
+
T_{\text{other}}.
]
Report how those components change as you increase the number of GPUs.
If communication becomes, say, 40% of the iteration at 32 GPUs, that’s not necessarily a flaw in your experiment. That is an important experimental result about the scalability of your particular hardware/network configuration.
Your biggest issue is probably the network
The Google Cloud configuration you describe has up to 24 Gbps per VM. That’s dramatically different from the bandwidth available through NVLink/NVSwitch inside a multi-GPU node.
Therefore, don’t simply say:
“We evaluated distributed training on 32 A100 GPUs.”
A more defensible description would be:
“We evaluated distributed training across 32 A100 40 GB GPUs deployed as one GPU per node, thereby studying a highly distributed, network-dependent configuration.”
Then explicitly report the network topology and measured bandwidth.
A useful control
If possible without increasing cost, don’t spend all 32 GPUs on a single configuration.
For example:
| Configuration |
GPUs |
Purpose |
| 1 node × 1 GPU |
1 |
Compute baseline |
| 2 nodes × 1 GPU |
2 |
Initial communication overhead |
| 4 nodes × 1 GPU |
4 |
Scaling |
| 8 nodes × 1 GPU |
8 |
Scaling |
| 16 nodes × 1 GPU |
16 |
Scaling |
| 32 nodes × 1 GPU |
32 |
Full experiment |
Even better, if the infrastructure allows it at roughly the same total GPU cost, compare:
32 nodes × 1 GPU
against something such as
16 nodes × 2 GPUs
or
8 nodes × 4 GPUs
That would directly demonstrate how much of your observed behavior comes from the number of GPUs versus the number of network hops/nodes.
One important caveat
Don’t call the configuration “unrealistic” simply because it has one GPU per node. There are legitimate distributed workloads and research questions where the network is precisely what you want to study.
The problem is generalization.
Your experiment can legitimately answer:
“How does this workload scale across 32 network-connected A100 GPUs under this network topology?”
It cannot, by itself, establish:
“This is how 32 A100 GPUs would perform in a conventional NVLink/NVSwitch cluster.”
So I would keep the setup if cost is the constraint. Just characterize the topology carefully, benchmark NCCL separately, report communication/computation breakdown, and make the scope of your claims explicit.
One additional point: if this is intended for a paper, the strongest design would be to make the network limitation itself an experimental variable, rather than treating it as something to hide. That turns a potential weakness of the hardware into a clearly defined experimental condition.