How to correctly use a normal distribution?

Let’s look at this small code:

import torch
import torch.nn as nn
from torch.distributions import Normal

class Proba(nn.Module):
    def __init__(self, size):
        super(Proba, self).__init__() = nn.Linear(size, size)
    def forward(self, obs):
        mu =
        dist = Normal(mu, 0.2)
        log_prob = dist.log_prob(obs)
        return log_prob.mean()

proba = Proba(10)
optimizer = torch.optim.Adam(proba.parameters(), lr=1e-3)

# fixed observation to reconstruct:
obs = torch.rand(10)

# want to optimize the probability to reconstruct obs:
if __name__=="__main__":

    for i in range(500):
        loss = -proba(obs)

        if i%100==0:
            # should never be larger than 0:
            print("log_prob of obs =", -loss.item())

The goal of this code is to reconstruct a signal “obs” by training a linear model that overfits an element-wise normal distribution with a fixed small standard deviation (0.2).

However, after ~500 iterations, the predicted log_prob starts being positive, while log_probabilites should never be positive.

Is something wrong in the code and the utilization of distributions.Normal?

Okay, I’ve just been through the source code and understood that it’s actually the log of the density, and not the log of a probability.

It makes sense since the measure of the event is null. I was rather expecting the proba of being in the interval [mu-|obs|, mu+|obs|]

Hi @alexis-jacq by looking at your question, is this a way to fit probabilistic models in pytorch, say estima the mean and variance for a normal likelyhood regression model?

@alexis-jacq Basically, I am trying to replicate this code from Keras to Pytorch:

def NLL(y, distr):
  return -distr.log_prob(y) #A

def my_dist(params):
  return tfd.Normal(loc=params, scale=1)

# set the sd to the fixed value 1
inputs = Input(shape=(1,))
params = Dense(1)(inputs)
dist = tfp.layers.DistributionLambda(my_dist)(params)

model_sd_1 = Model(inputs=inputs, outputs=dist)
model_sd_1.compile(Adam(), loss=NLL)

Here, we set up a (minimal) probabilistic neural network as a regression.

Any ideas how in PyTorch?