Why .eval() is missing from this official Pytorch tutorial?

John_Deterious · July 24, 2019, 7:02pm

Here’s a DCGAN tutotial
https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html

When evaluating the performance, .eval() and .train() is completely ignored, no mention of them in the entire code. Why is that? Thanks.

EDIT: here, I can point to the exact line where I think this is missing

github.com

pytorch/tutorials/blob/master/beginner_source/dcgan_faces_tutorial.py#L646


        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))
        
        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())
        
        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))
            
        iters += 1




######################################################################
# Results

Balamurali_M · July 25, 2019, 2:52am

I have also noticed this. Even while I code, even if I miss eval and train. It didn’t have any effect on the result.

But the thing is how we are going about it does matter. We use eval because we won’t be interested in updating the weight of the network. While train weight updation happens. If we can turn off the gradients using torch no grad like your example, be it. It’s another way of approaching I think.

Swarchal · July 25, 2019, 7:06am

My understanding is that .eval() is to tell the network to disable dropout and batchnorm layers, where as the torch.no_grad() context is to disable gradient calculations. They are different concepts which happen to be used together during inference.

One possibility why .eval() is missing is because there are no dropout or batchnorm layers?

Balamurali_M · July 25, 2019, 3:44pm

Dropout, batchnorm also seems like a proper answer. Can you let us know about that ?

John_Deterious · July 26, 2019, 8:57am

There are. In both generator and discriminator.

John_Deterious · July 26, 2019, 8:58am

It does have an effect. If I take 1000 examples from the generator, the images get whiter and whiter. To fix the performance I go to eval() mode.