Difficulty understanding entropy() in PyTorch

1

I’m new to PyTorch, and I’m having trouble interpreting entropy. Suppose, we have a probability distribution [0.1, 0.2, 0.4, 0.3]

First, let’s calculate entropy using numpy.

import numpy as np

p = np.array([0.1, 0.2, 0.4, 0.3])
logp = np.log2(p)
entropy1 = np.sum(-p*logp)
print(entropy1)

Output:

1.846439

Next, let’s use entropy() from torch.distributions.Categorical

import torch
from torch.distributions import Categorical

p_tensor = torch.Tensor([0.1, 0.2, 0.4, 0.3])
entropy2 = Categorical(probs = p_tensor).entropy()
print(entropy2)

Output:

tensor(1.2799)

Why is entropy1 not equal to entropy2?

6 Likes
2

Hi kabron_wade,
The entropy is calculated using the natural logarithm. In your numpy example code, you use np.log2(). Using np.log() would give you the same result as the pytorch entropy().

10 Likes