# How to set OMP\_NUM\_THREADS for distruted training?

**URL:** <https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234>\
**Category:** Uncategorized\
**Created:** [February 3, 2022, 11:16pm UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234 "2022-02-03T23:16:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [February 3, 2022, 11:16pm UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/1 "2022-02-03T23:16:20Z")

</div>

I got a warning but there was no link or suggestions of how to tune this number (or what it means). How do I choose this value?

Error:

```auto
(meta_learning_a100) [miranda9@hal-dgx diversity-for-predictive-success-of-meta-learning]$ python -m torch.distributed.launch --nproc_per_node=2 ~/ultimate-utils/tutorials_for_myself/my_l2l/dist_maml_l2l_from_seba.py

/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torch.distributed.run.
Note that --use_env is set by default in torch.distributed.run.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
WARNING:torch.distributed.run: *****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************

```

---

<div class="post-metadata">

**Author:** ![Dazitu616](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dazitu616/32/39023_2.png) [@Dazitu616](https://discuss.pytorch.org/u/Dazitu616)\
**Post date:** [February 4, 2022, 1:58am UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/2 "2022-02-04T01:58:35Z")

</div>

Since it’s an environment variable, I think you can simply set its value by:  
`OMP_NUM_THREADS=$VALUE python -m torch.distributed.launch --nproc_per_node=2 xxxxx`  
This is similar to other environment variables e.g. `CUDA_VISIBLE_DEVICES`

---

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [February 4, 2022, 2:13am UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/3 "2022-02-04T02:13:34Z")

</div>

oh cool. But I was curious about how does one choose the value of `OMP_NUM_THREADS=$VALUE`

---

<div class="post-metadata">

**Author:** ![likaizhou8](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/likaizhou8/32/60128_2.png) [@likaizhou8](https://discuss.pytorch.org/u/likaizhou8)\
**Post date:** [May 2, 2023, 3:51pm UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/4 "2023-05-02T15:51:10Z")

</div>

bro，did you have slove this problem?

---

<div class="post-metadata">

**Author:** ![Kenzo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kenzo/32/10615_2.png) [@Kenzo](https://discuss.pytorch.org/u/Kenzo)\
**Post date:** [February 12, 2024, 12:15pm UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/5 "2024-02-12T12:15:25Z")

</div>

It seems that an optimal value can be found by OMP\_NUM\_THREADS = nb\_cpu\_threads / nproc\_per\_node. Use htop to know about number of CPU threads

---

<div class="post-metadata">

**Author:** ![handaru](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/handaru/32/75558_2.png) [@handaru](https://discuss.pytorch.org/u/handaru)\
**Post date:** [February 10, 2025, 12:44pm UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/6 "2025-02-10T12:44:15Z")

</div>

# set omp threads to number of logical cpus

import os

logical\_cpus = os.process\_cpu\_count()

print(logical\_cpus)

os.environ[‘OMP\_NUM\_THREADS’] = str(logical\_cpus)

---

<div class="post-metadata">

**Author:** ![MinSnz](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minsnz/32/76345_2.png) [@MinSnz](https://discuss.pytorch.org/u/MinSnz)\
**Post date:** [March 17, 2025, 2:10pm UTC](https://discuss.pytorch.org/t/how-to-set-omp-num-threads-for-distruted-training/143234/7 "2025-03-17T14:10:22Z")

</div>

`OMP_NUM_THREADS` should be set to `#PhysicalCores / #Processes`.

- `os.process_cpu_count`, as @handaru mentioned and was [added in Python 3.13](https://docs.python.org/3/library/os.html#os.process_cpu_count), as well as `nproc` and `htop`, gives the number of _logical_ CPUs. Please use `psutil.cpu_count(logical=False)` to get the number of **physical** CPUs.
- Don’t forget to divide it by `nproc_per_node` as @Kenzo mentioned, or you’ll have many processes competing for each core.