Normal distribution: A log_prob dimension problem

Hello,

I’m trying to work my way through Mixtures Density Network, and while I managed to figure it out for a single feature for the output, I’m stuck when it comes to several features. Among my problems, here’s the one concerning the Normal distribution class.

Here’s an example:

import torch
from torch.distributions import Normal 

means = torch.zeros(1,2)
stds = torch.ones(1,2)

dist = Normal(means, stds)

print('Sample is size {}'.format(dist.sample().shape))
print('Log prob of {} = {}'.format(torch.zeros(1,2), dist.log_prob(torch.zeros(1,2))))

Outputs:

Sample is size torch.Size([1, 2])
Log prob of tensor([[ 0.,  0.]]) = tensor([[-0.9189, -0.9189]])

Shouldn’t the log probability be of a single dimension ? Am I (certainly but what) doing something wrong ?

Thanks a lot !

Never mind, I figured it out !
I think that Normal generates one normal distribution per feature. Hence, you have as many log_probs as you have distributions.
However, in the case of multi-features, here’s what I propose:

  • Either keep Normal, but sum log probs (or multiply probs) to get the probability you seek given multifeatures
  • Use a Multivariate Normal, which avoids you to gather the probabilites yourself.

If anyone has other insights, or if I’ve made a mistake, I’d be delighted to hear it !

Thanks !

1 Like

dist.log_prob(a) should have kept the same dimension as a. The essence of dist.log_prob() can be explained as the following example:

prob = torch.rand(5)
m=Bernoulli(prob)
act = m.sample()

act
tensor([0., 0., 1., 0., 0.])

prob
tensor([0.4880, 0.5403, 0.1671, 0.1158, 0.1695])

m.log_prob(act)
tensor([-0.6694, -0.7773, -1.7889, -0.1230, -0.1857])

torch.log(prob)
tensor([-0.7175, -0.6155, -1.7889, -2.1562, -1.7751])

torch.log(1-prob)
tensor([-0.6694, -0.7773, -0.1829, -0.1230, -0.1857])

So, why, in your example, for torch.zeros(1,2), you said that the log_prob is a single value instead of being with size (1,2)?