How to correctly use a normal distribution?

1

Let’s look at this small code:

import torch
import torch.nn as nn
from torch.distributions import Normal

class Proba(nn.Module):
    def __init__(self, size):
        super(Proba, self).__init__()
        self.mu = nn.Linear(size, size)
        
    def forward(self, obs):
        mu = self.mu(obs)
        dist = Normal(mu, 0.2)
        log_prob = dist.log_prob(obs)
        return log_prob.mean()

proba = Proba(10)
optimizer = torch.optim.Adam(proba.parameters(), lr=1e-3)

# fixed observation to reconstruct:
obs = torch.rand(10)

# want to optimize the probability to reconstruct obs:
if __name__=="__main__":

    for i in range(500):
        loss = -proba(obs)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if i%100==0:
            # should never be larger than 0:
            print("log_prob of obs =", -loss.item())

The goal of this code is to reconstruct a signal “obs” by training a linear model that overfits an element-wise normal distribution with a fixed small standard deviation (0.2).

However, after ~500 iterations, the predicted log_prob starts being positive, while log_probabilites should never be positive.

Is something wrong in the code and the utilization of distributions.Normal?

2

Okay, I’ve just been through the source code and understood that it’s actually the log of the density, and not the log of a probability.

It makes sense since the measure of the event is null. I was rather expecting the proba of being in the interval [mu-|obs|, mu+|obs|]

3

Hi @alexis-jacq by looking at your question, is this a way to fit probabilistic models in pytorch, say estima the mean and variance for a normal likelyhood regression model?

4

@alexis-jacq Basically, I am trying to replicate this code from Keras to Pytorch:

def NLL(y, distr):
  return -distr.log_prob(y) #A

def my_dist(params):
  return tfd.Normal(loc=params, scale=1)

# set the sd to the fixed value 1
inputs = Input(shape=(1,))
params = Dense(1)(inputs)
dist = tfp.layers.DistributionLambda(my_dist)(params)

model_sd_1 = Model(inputs=inputs, outputs=dist)
model_sd_1.compile(Adam(), loss=NLL)

Here, we set up a (minimal) probabilistic neural network as a regression.

Any ideas how in PyTorch?