import torch
from torch import nn
from torch.autograd import Variable
torch.manual_seed(1)
net = nn.Sequential(
nn.Linear(100, 100),
nn.BatchNorm1d(100),
nn.Linear(100, 1)
)
im = Variable(torch.ones(10, 100))
for _ in range(10):
net.train(True)
print(net(im))
for _ in range(10):
net.train(False)
print(net(im))
the first 10 prints output arrays with one value(-8.6313 * 1e-2
), and the second 10 prints output arrays with another value (-0.1563 * 1e-2
).
It is rather strange and confusing.
-
the first 10 output is the same, is it reasonable? I think the
runing_mean
andrunning_var
have some initial value and then get updated at each step. so the output should be changing. -
the second 10 output is the same, it is reasonable since it’s in eval mode. what is confusing is that it is not the same as the last training output. I think if we set the model to evaluation mode, it just freezes
runing_mean
andrunning_var
, which should make the second 10 outputs the same as the last training output.
Any ideas? I’m almost driven crazy by this!