Hi there,
I am planning on using the Batch Norm layer between the two other layers of my model. After training the model I am not sure how to 'un’normalize the output when I want to receive the predicted values?
If it would be simply normalization on some values - I would just calculate and save the variance and mean, then use these values for 'un’normalizing the output of the model - when I want to get the predictions.
In my view, in the case of batch normalization - since the normalization is done based on the batches and these batches are different at every epoch, these mean and variance values will be different at each epoch - so I dont see a way to save those values for using it for unnormalizing the predicted values…
To get the predictions of the model (assuming you are using a validation or test dataset), you would not “unnormalize” the data, but would use the running stats instead by calling model.eval().
During training the internal stats (stored in .running_mean and .running_var in batchnorm layers) will be updated with a specified momentum and the current batch stats.
@ptrblck Thank you for your reply.
I had one more question regarding the topic following your answer:
Let’s suppose that my forward method looks like this:
def forward(self, x, hs):
out, hs = self.lstm(x, hs)
out = self.batchnormalization(out)
out = self.fullyconnected(out)
return out, hs
Then in my training and validation code:
for e in range(epochs):
training_loss = 0
validation_loss = 0
model.train()
for x,y training_data:
train_h = model.init_hidden(batch_size)
train_h = tuple([h.data for h in train_h])
out, train_h = model(x, train_h)
**loss = criterion(out,y)**
training_loss += loss.item()
opt.zero_grad()
loss.backward()
optimizer.step()
training_losses.append(training_loss/len(training_data))
# validation
model.eval()
with torch.no_grad():
for x, y in validation_data:
val_h = model.init_hidden(batch_size)
val_h = tuple([h.data for h in val_h])
out, val_h = model(x, val_h)
**loss = criterion(out, y)**
validation_loss += loss.item()
validation_losses.append(validation_loss/len(validation_data))
Then in my training and validation code how should I use the running stats that you mentioned? Since as far as I understand, I cant just do loss = criterion(out, y), right? (because the ‘out’ was calculated using the BatchNorm)
The model.train() and model.eval() calls will switch between the training mode (normalizing the input batch with its own stats and updating the running stats) and evaluation mode (normalizing the input batch/sample with the running stats). You don’t need to apply the running stats manually.