Why running an extra forward function with random generated input but no loss calculation or backpropagation in the previous normal training process can change the training result?

vision
1

I am trying a new training method (semi supervise GAN) but there is some result I don’t understand. The problem seems to be some Pytorch feature I don’t understand, but not in semi supervise GAN.
The classifier is the EfficientNet_b0.

for epoch in range(1, epochs + 1):
print (’\nEpoch: {} '.format(epoch))

for idx, dataLabeled in enumerate(train_loader):
    classifier.train()
    optimizerC.zero_grad()

    # I am working on these 3 lines that cause strange results
    # These 3 lines is a small part of some semi supervise GAN but not completed.    
    noise = torch.randn(BATCH_SIZE, 100, 1, 1, device=device)
    generated = (netG(noise)+1.0)/2.0
    # outputG = classifier(generated.detach())    

    labeled_data, labels = dataLabeled
    labeled_data, labels = labeled_data.to(device), labels.to(device)

    output = classifier(labeled_data)
    labeled_loss = criterion(output, labels) 
    labeled_loss.backward()    
    optimizerC.step()

correct = 0
num_test = len(test_loader.dataset)
# testing accuracy, I don't think the problem is here 
with torch.no_grad():
    for batch in test_loader:
        image, labelS = batch
        data = image.to(device)
        target = labelS.to(device)
        output = model(data)
        test_loss += criterion(output, target).item() 
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()
    print('\nTest set: Accuracy: {}/{} ({:.4f}%)\n'.format(correct, num_test, 100. * correct / num_test))

After I commented out the line # outputG = classifier(generated.detach()), it worked just like the EfficientNet. I tested it several times and the results were similar.
Epoch: 1 Test set: Accuracy: 114/163 (69.9387%)
Epoch: 2 Test set: Accuracy: 123/163 (75.4601%)
Epoch: 3 Test set: Accuracy: 131/163 (80.3681%)
Epoch: 4 Test set: Accuracy: 126/163 (77.3006%)

If I uncommented this line, the results changed. The accuracy keeps going down every time I try. The only difference is adding this line # outputG = classifier(generated.detach()).
Epoch: 1 Test set: Accuracy: 99/163 (60.7362%)
Epoch: 2 Test set: Accuracy: 117/163 (71.7791%)
Epoch: 3 Test set: Accuracy: 106/163 (65.0307%)
Epoch: 4 Test set: Accuracy: 97/163 (59.5092%)

So why this one line outputG = classifier(generated.detach()) can make the result changed? It’s just a forward function. I don’t even run loss calculation or backpropagation for it.

2

Some layer, such as batchnorm layers, will perform updates of stats during the forward pass.
In the example of batchnorm layers (which are used in EfficientNet_b0), the running stats will be updated using the current batch statistics and will thus change the validation results.

1 Like