Why running an extra forward function with random generated input but no loss calculation or backpropagation in the previous normal training process can change the training result?

I am trying a new training method (semi supervise GAN) but there is some result I don’t understand. The problem seems to be some Pytorch feature I don’t understand, but not in semi supervise GAN.
The classifier is the EfficientNet_b0.

for epoch in range(1, epochs + 1):
print (’\nEpoch: {} '.format(epoch))

for idx, dataLabeled in enumerate(train_loader):

    # I am working on these 3 lines that cause strange results
    # These 3 lines is a small part of some semi supervise GAN but not completed.    
    noise = torch.randn(BATCH_SIZE, 100, 1, 1, device=device)
    generated = (netG(noise)+1.0)/2.0
    # outputG = classifier(generated.detach())    

    labeled_data, labels = dataLabeled
    labeled_data, labels = labeled_data.to(device), labels.to(device)

    output = classifier(labeled_data)
    labeled_loss = criterion(output, labels) 

correct = 0
num_test = len(test_loader.dataset)
# testing accuracy, I don't think the problem is here 
with torch.no_grad():
    for batch in test_loader:
        image, labelS = batch
        data = image.to(device)
        target = labelS.to(device)
        output = model(data)
        test_loss += criterion(output, target).item() 
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()
    print('\nTest set: Accuracy: {}/{} ({:.4f}%)\n'.format(correct, num_test, 100. * correct / num_test))

After I commented out the line # outputG = classifier(generated.detach()), it worked just like the EfficientNet. I tested it several times and the results were similar.
Epoch: 1 Test set: Accuracy: 114/163 (69.9387%)
Epoch: 2 Test set: Accuracy: 123/163 (75.4601%)
Epoch: 3 Test set: Accuracy: 131/163 (80.3681%)
Epoch: 4 Test set: Accuracy: 126/163 (77.3006%)

If I uncommented this line, the results changed. The accuracy keeps going down every time I try. The only difference is adding this line # outputG = classifier(generated.detach()).
Epoch: 1 Test set: Accuracy: 99/163 (60.7362%)
Epoch: 2 Test set: Accuracy: 117/163 (71.7791%)
Epoch: 3 Test set: Accuracy: 106/163 (65.0307%)
Epoch: 4 Test set: Accuracy: 97/163 (59.5092%)

So why this one line outputG = classifier(generated.detach()) can make the result changed? It’s just a forward function. I don’t even run loss calculation or backpropagation for it.

Some layer, such as batchnorm layers, will perform updates of stats during the forward pass.
In the example of batchnorm layers (which are used in EfficientNet_b0), the running stats will be updated using the current batch statistics and will thus change the validation results.

1 Like