Loss abnormal after using batchnorm1d

autograd
1

Hi, I’m training a model with BCEWithLogitsLoss. The model has a head like this:

self.head = nn.Linear(config.hidden_size, num_labels)

My loss during this training is about 0.3, when I use the head below:

self.head = nn.Sequential(  #
            nn.Linear(config.hidden_size, 150),
            nn.BatchNorm1d(150),
            nn.Linear(150, num_labels),
        )

The loss goes to 0.6. Is this a normal situation?

2

Is just the initial loss higher?
How does the training look like? Are you able to get the same or lower loss with the batchnorm layer or does it stay at a higher level constantly?

3

The training process is roughly as follows:

58
582244×1366 474 KB
53
532236×1206 413 KB

The first one uses BN, the second one does not use BN. The training code is long, I send you a message with it. Thank you.

4

While the training loss seems to be higher, the validation score seems to be better in the batchnorm model.
In that case, I would stick to it and maybe play around with some hyperparameters.

5

Yes, the batchnorm model has better validation score, but worse loss. Is it because of the multi-gpu?

6

It’s hard to tell, where exactly this effect comes from and I have not really any idea, so let’s wait for some experts on this topic :slight_smile: