Normal distribution: A log_prob dimension problem


I’m trying to work my way through Mixtures Density Network, and while I managed to figure it out for a single feature for the output, I’m stuck when it comes to several features. Among my problems, here’s the one concerning the Normal distribution class.

Here’s an example:

import torch
from torch.distributions import Normal 

means = torch.zeros(1,2)
stds = torch.ones(1,2)

dist = Normal(means, stds)

print('Sample is size {}'.format(dist.sample().shape))
print('Log prob of {} = {}'.format(torch.zeros(1,2), dist.log_prob(torch.zeros(1,2))))


Sample is size torch.Size([1, 2])
Log prob of tensor([[ 0.,  0.]]) = tensor([[-0.9189, -0.9189]])

Shouldn’t the log probability be of a single dimension ? Am I (certainly but what) doing something wrong ?

Thanks a lot !

Never mind, I figured it out !
I think that Normal generates one normal distribution per feature. Hence, you have as many log_probs as you have distributions.
However, in the case of multi-features, here’s what I propose:

  • Either keep Normal, but sum log probs (or multiply probs) to get the probability you seek given multifeatures
  • Use a Multivariate Normal, which avoids you to gather the probabilites yourself.

If anyone has other insights, or if I’ve made a mistake, I’d be delighted to hear it !

Thanks !

1 Like