# What does “normal random sampling” mean in LLM text generation?

**URL:** <https://discuss.pytorch.org/t/what-does-normal-random-sampling-mean-in-llm-text-generation/225384>\
**Category:** Uncategorized\
**Created:** [September 7, 2026, 1:00pm UTC](https://discuss.pytorch.org/t/what-does-normal-random-sampling-mean-in-llm-text-generation/225384 "2026-09-07T13:00:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![nono345](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nono345/32/80004_2.png) [@nono345](https://discuss.pytorch.org/u/nono345)\
**Post date:** [September 7, 2026, 1:00pm UTC](https://discuss.pytorch.org/t/what-does-normal-random-sampling-mean-in-llm-text-generation/225384/1 "2026-09-07T13:00:57Z")

</div>

I am working on research involving the numerical behavior of LLM token sampling, and I need to reproduce the ordinary stochastic next-token sampling process as faithfully as possible.

At each decoding step, I obtain the model logits and convert them to probabilities:

```python
logits = model(...).logits[:, -1, :]
probs = torch.softmax(logits, dim=-1)

```

I can then sample the next token in at least two ways.

Using PyTorch:

```python
next_token = torch.multinomial(probs, num_samples=1)

```

or by explicitly implementing inverse-transform sampling:

```python
cdf = torch.cumsum(probs, dim=-1)
u = torch.rand(1, device=probs.device)
next_token = torch.searchsorted(cdf, u)

```

Mathematically, both are intended to sample from the same categorical distribution:

$$  
X \sim \mathrm{Categorical}(p).  
$$

However, my question is specifically about the **concrete finite-precision implementation** , not only the mathematical distribution.

With a large LLM vocabulary and `float32` probabilities, I have observed cases where the explicitly accumulated CDF contains adjacent equal values:

```python
cdf[i] == cdf[i - 1]

```

even though:

```python
probs[i] > 0

```

In that case, the interval width for token `i` becomes zero in this particular inverse-CDF implementation.

I have also generated tokens using:

```python
torch.multinomial(probs, 1)

```

and observed behavior that is not always reproduced by my explicit FP32 CDF sampler.

I checked the PyTorch documentation, but it appears to describe the intended sampling distribution rather than guarantee that `torch.multinomial(probs, 1)` is implemented as the same serial `cumsum + uniform random number + searchsorted` procedure shown above.

So my questions are:

1. For `torch.multinomial(probs, 1)`, what sampling algorithm does PyTorch actually use?

2. Is this implementation:

```python
cdf = torch.cumsum(probs.float(), dim=-1)
u = torch.rand(1, device=probs.device)
sample = torch.searchsorted(cdf, u)

```

expected to have the same **finite-precision sampling behavior** as:

```python
torch.multinomial(probs, 1)

```

or are they only equivalent in exact arithmetic?

1. In PyTorch-based LLM generation, when someone refers to “random sampling” or “sampling from the model distribution,” is `torch.multinomial(probs, 1)` the usual concrete implementation, or is the phrase intended only at the abstract distribution level?

2. If I want to reproduce the ordinary stochastic sampling behavior of a PyTorch LLM implementation, should I use the framework’s native sampler (`torch.multinomial`) rather than assume that “random sampling” means a particular inverse-CDF implementation?