# \`sm\_89\` not listed in the \`torch.cuda.get\_arch\_list()\`

**URL:** <https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827>\
**Category:** Uncategorized\
**Created:** [January 24, 2025, 2:41pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827 "2025-01-24T14:41:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vgoklani](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vgoklani/32/1971_2.png) [@vgoklani](https://discuss.pytorch.org/u/vgoklani)\
**Post date:** [January 24, 2025, 2:41pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827/1 "2025-01-24T14:41:00Z")

</div>

We’re getting the docker image from here: `nvcr.io/nvidia/pytorch:24.12-py3`

and when we get the arch list:

```python
import torch
torch.cuda.get_arch_list()

# ['sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_90']

torch.version.cuda

# '12.6'

```

`sm_89` is not listed. We are running 4x NVIDIA RTX 6000 Ada cards.

We also tried installing the latest version of torch via pip, but still don’t see `sm_89` listed. Do we have to do something special to enable it?

Here is the CUDA information if needed:

```bash
root@~ $ nvidia-smi --version
NVIDIA-SMI version : 565.57.01
NVML version : 565.57
DRIVER version : 565.57.01
CUDA Version : 12.7

```

Thanks!

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [January 24, 2025, 3:30pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827/2 "2025-01-24T15:30:56Z")

</div>

> [@vgoklani](#):
>
> Do we have to do something special to enable it?

No, you don’t need to build for `sm_89`as it’s binary compatible with `sm_86/sm_80`. Your device is thus supported in all of our builds.

---

<div class="post-metadata">

**Author:** ![vgoklani](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vgoklani/32/1971_2.png) [@vgoklani](https://discuss.pytorch.org/u/vgoklani)\
**Post date:** [January 24, 2025, 3:44pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827/3 "2025-01-24T15:44:26Z")

</div>

Thanks @ptrblck so there shouldn’t be any issues running FP8 specific kernels in pytorch? Those aren’t part of sm\_86…

More specifically I’m unable to utilize row-wise scaling in FP8

> <https://github.com/pytorch/ao/issues/1057#issuecomment-2610796124>
>
> I am running torchao: 0.5 and torch: '2.5.0a0+b465a5843b.nv24.09' on an NVIDIA A…6000 ADA card (sm89) which supports FP8.
> 
> I ran the generate.py code from the benchmark:
> 
> python generate.py --checkpoint\_path $CHECKPOINT\_PATH --compile --compile\_prefill --write\_result /root/benchmark\_results\_\_baseline.txt
> 
> \> Average tokens/sec: 57.01
> \> Average Bandwidth: 855.74 GB/s
> \> Peak Memory Usage: 16.19 GB
> \> Model Size: 15.01 GB
> 
> \> 20241011143042, tok/s= 57.01, mem/s= 855.74 GB/s, peak\_mem=16.19 GB, model\_size=15.01 GB quant: None, mod: Meta-Llama-3-8B, kv\_quant: False, compile: True, compile\_prefill: True, dtype: torch.bfloat16, device: cuda repro: python generate.py --checkpoint\_path /models/Meta-Llama-3-8B/consolidated.00.pth --device cuda --precision torch.bfloat16 --compile --compile\_prefill --num\_samples 5 --max\_new\_tokens 200 --top\_k 200 --temperature 0.8
> 
> python generate.py --checkpoint\_path $CHECKPOINT\_PATH --compile --compile\_prefill --quantization float8wo --write\_result /root/benchmark\_results\_\_float8wo.txt\`
> 
> \> Average tokens/sec: 57.00
> \> Average Bandwidth: 855.62 GB/s
> \> Peak Memory Usage: 16.19 GB
> \> Model Size: 15.01 GB
> 
> \> 20241011143316, tok/s= 57.00, mem/s= 855.62 GB/s, peak\_mem=16.19 GB, model\_size=15.01 GB quant: float8wo, mod: Meta-Llama-3-8B, kv\_quant: False, compile: True, compile\_prefill: True, dtype: torch.bfloat16, device: cuda repro: python generate.py
> \--quantization float8wo --checkpoint\_path /models/Meta-Llama-3-8B/consolidated.00.pth --device cuda --precision torch.bfloat16 --compile --compile\_prefill --num\_samples 5 --max\_new\_tokens 200 --top\_k 200 --temperature 0.8
> 
> The \`float8wo\` flag does not appear to be doing anything. Am I missing a step? Thanks!

The response from the thread was that sm\_89 should be listed in the arch\_list.

Thanks!

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [January 24, 2025, 6:32pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827/4 "2025-01-24T18:32:51Z")

</div>

I don’t know if the current row-wise scaling kernel implementation is compatible with `sm_89` and we are explicitly building them for `sm_90a` (arch-conditional) [here](https://github.com/pytorch/pytorch/blob/0f051eaf66b5045fe0131e75966be2cb7645687b/cmake/Codegen.cmake#L80-L92).

---

<div class="post-metadata">

**Author:** ![vgoklani](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vgoklani/32/1971_2.png) [@vgoklani](https://discuss.pytorch.org/u/vgoklani)\
**Post date:** [January 24, 2025, 8:21pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827/5 "2025-01-24T20:21:18Z")

</div>

There was a recent PR - [Add SM89 support for f8f8bf16\_rowwise() by alexsamardzic · Pull Request #144348 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/pull/144348)

which “introduced support for `_scaled_mm` operator with FP8 inputs on SM89 architecture. The support is based on CUTLASS library, that is header-only C++ library, so this new functionality gets fully built along with PyTorch build; however, it will get built only in case the build includes SM89 among targets.”

Unfortunately this requires that `sm_89` is on the list of targets.

I just opened a ticket [Support `sm_89` in Stable/Nightly/Docker Images · Issue #145632 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/145632)

Does this make sense?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [January 24, 2025, 8:32pm UTC](https://discuss.pytorch.org/t/sm-89-not-listed-in-the-torch-cuda-get-arch-list/215827/6 "2025-01-24T20:32:06Z")

</div>

No, we should not add `sm_89` directly as it will waste space with no benefits besides FP8 support. Instead, we should add `sm_89` to the one file supporting FP8 as explained in my comment on GitHub.
