# Difference between torch.distributed and python run

**URL:** <https://discuss.pytorch.org/t/difference-between-torch-distributed-and-python-run/150494>\
**Category:** distributed\
**Created:** [April 30, 2022, 7:44pm UTC](https://discuss.pytorch.org/t/difference-between-torch-distributed-and-python-run/150494 "2022-04-30T19:44:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![base-y](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d2c977/32.png) [@base-y](https://discuss.pytorch.org/u/base-y)\
**Post date:** [April 30, 2022, 7:44pm UTC](https://discuss.pytorch.org/t/difference-between-torch-distributed-and-python-run/150494/1 "2022-04-30T19:44:04Z")

</div>

Hi, can I know whats the difference between the following:

1. `python train.py .....<ARGS>`
2. `python -m torch.distributed.launch <ARGS>`
3. `deepspeed train.py <ARGS>`

Even option 1 seem to be using some sort of distributed training when there are multiple gpus. My understanding was option 1 will only use 1 gpu if not given `distributed.launch`.

Also, does deepspeed use torch.distributed in the backend or is it something different?

thanks in advance

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 2, 2022, 7:12am UTC](https://discuss.pytorch.org/t/difference-between-torch-distributed-and-python-run/150494/2 "2022-05-02T07:12:26Z")

</div>

> [@base-y](#):
>
> Even option 1 seem to be using some sort of distributed training when there are multiple gpus. My understanding was option 1 will only use 1 gpu if not given `distributed.launch`.

It depends what `train.py` uses as inside the script still multiple processes could be spawned.

---

<div class="post-metadata">

**Author:** ![cbalioglu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cbalioglu/32/33187_2.png) [@cbalioglu](https://discuss.pytorch.org/u/cbalioglu)\
**Post date:** [May 2, 2022, 6:23pm UTC](https://discuss.pytorch.org/t/difference-between-torch-distributed-and-python-run/150494/3 "2022-05-02T18:23:23Z")

</div>

In addition to @ptrblck’s answer; there is really no magic inside `python -m torch.distributed.launch`. It can be considered mostly a convenience launcher script that helps you to run distributed jobs while avoiding most of the boilerplate code. Ultimately it set ups the process group(s), environment variables, etc. that you can consume in your training script passed in `<ARGS>`. Technically there is nothing preventing you from replicating all that code in a custom `train.py` script (i.e. option 1), but in most use cases using `python -m torch.distributed.launch` is much easier and future-proof.

---

<div class="post-metadata">

**Author:** ![base-y](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/d2c977/32.png) [@base-y](https://discuss.pytorch.org/u/base-y)\
**Post date:** [May 6, 2022, 1:10pm UTC](https://discuss.pytorch.org/t/difference-between-torch-distributed-and-python-run/150494/4 "2022-05-06T13:10:55Z")

</div>

> [@ptrblck](#):
>
> It depends what `train.py` uses as inside the script still multiple processes could be spawned.

I am using huggingface trainer inside `train.py`. Could be that the trainer is spawning multiple processes. Maybe thats why `option 1` too is using dist.training.