Hi there!
I’m trying to use the pyro framework to learn the linear function f(x)=5x+2 - for this I’ve worked through the Bayesian Regression Tutorial.
First I’ve designed some noisy data points of the function with
x_data = torch.rand((500, 1))
y_data = 5*x_data +2. + torch.normal(0, 0.1, size=(500, 1))
Then I’ve build the Bayesian Regression Model. For a sanity check, I’ve set the priors of the weight and the bias to the true values and set a very low variance.
class BayesianRegression(PyroModule):
def __init__(self, in_features, out_features):
super().__init__()
self.linear = PyroModule[nn.Linear](in_features, out_features)
self.linear.weight = PyroSample(dist.Normal(5, 1).expand([out_features, in_features]).to_event(2))
self.linear.bias = PyroSample(dist.Normal(2, 1).expand([out_features]).to_event(1))
def forward(self, x, y=None):
sigma = pyro.sample("sigma", dist.Uniform(0., 1))
mean = self.linear(x).squeeze(-1)
with pyro.plate("data", x.shape[0]):
obs = pyro.sample("obs", dist.Normal(mean, sigma), obs=y)
return mean
Then I’ve ran the SVI for 1500 iterations:
model = BayesianRegression(1, 1)
guide = AutoDiagonalNormal(model)
adam = pyro.optim.Adam({"lr": 0.03})
svi = SVI(model, guide, adam, loss=Trace_ELBO())
pyro.clear_param_store()
for j in range(num_iterations):
# calculate the loss and take a gradient step
loss = svi.step(x_data, y_data)
if j % 100 == 0:
print("[iteration %04d] loss: %.4f" % (j + 1, loss / len(x_data)))
Evaluating this model x=torch.arange(0, 1, 0.01) gives the following plot:
After the training the mean of the weight is 0.0011 and the bias is 4.5674.
I do not understand why these Means do not correspond to the correct values after the training, although the Priors already contain the desired value? Of course if I decrease the learning rate, the deviation from 5 and 2 is smaller after training, but this is obviously not the right way, because this only works because I already have the correct values in the prior.
Moreover if I e.g. choose other prios like
self.linear.weight = PyroSample(dist.Normal(0, 10).expand([out_features, in_features]).to_event(2))
self.linear.bias = PyroSample(dist.Normal(0, 10).expand([out_features]).to_event(1))
no learning effect occurs,even after 10.000 training iterations, although the variance in the priors is pretty high.
What am I doing wrong?