DCGAN - Trying to backward through the graph a second time

[KFrank]

Hi Uygar!

@dev1

Tried .detach(). and retain_graph=True options.

As a general principle, don’t just try things. Analyze the actual cause
of your specific issue and use the “option” that actually addresses that
cause.

Please look at the comments that I’ve added in line to your quoted code:

criterion = nn.BCELoss()

discriminator_optimizer = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
generator_optimizer = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

img_list = []
G_losses = []
D_losses = []
iters = 0

start = timeit.default_timer()
for epoch in tqdm(range(EPOCHS), position=0, leave=True):
    generator.train()
    discriminator.train()
    for idx, data in enumerate(tqdm(dataloader, position=0, leave=True)):
        img_data = data[0].to(device) # size [1024, 3, 64, 64]
        dummy_labels = data[1] # size [1024]
        
        real_labels = torch.full((dummy_labels.size()), 1., dtype=torch.float).to(device) # size [1024]
        fake_labels = torch.full((dummy_labels.size()), 0., dtype=torch.float).to(device)
        noise = torch.randn(dummy_labels.size()[0], INPUT_VECTOR_DIM, 1, 1).to(device) # size [1024, 100, 1, 1]
        
        discriminator.zero_grad() 
        
        discriminator_real_out = discriminator(img_data).view(-1).to(device) # size [1024] .view(-1) to rid unnecessary dimensions
        discriminator_real_loss = criterion(discriminator_real_out, real_labels)
        discriminator_real_loss.backward()
        
        generator_fake_out = generator(noise)  # size [1024, 3, 64, 64]
        discriminator_fake_out = discriminator(generator_fake_out.detach()).view(-1).to(device)
        ##### at this point, discriminator is included in the computation graph for discriminator_fake_out
        ##### note that discriminator_fake_out's computation graph does not include generator (because you called .detach() on generator_fake_out)
        discriminator_fake_loss = criterion(discriminator_fake_out, fake_labels)
        discriminator_fake_loss.backward()
        ##### this frees (most of) the computation graph that runs through discriminator
        ##### note, if you call .backward (retain_graph = True), you won't free the graph
        discriminator_loss = discriminator_real_loss.item() + discriminator_fake_loss.item()
        discriminator_optimizer.step()

        generator_loss = criterion(discriminator_fake_out, real_labels)
        ##### the computation graph for generator_loss contains discriminator_fake_out and therefore contains the mostly-freed graph for discriminator
        generator_loss.backward()
        ##### you now try to backpropagate through discriminator's mostly-freed graph, causing the "backward a second time" error
        generator_optimizer.step()
        ##### leaving aside the "backward a second time"" error, this won't do anything because you called .detach() on generator_fake_out, preventing any of the .backward() calls from populating generator's parameters with gradients
        generator.zero_grad()

The key issue is that you are using discriminator_fake_out to
optimize both discriminator and generator (leading in this particular
attempt to the “backward a second time” error). If you try using
retain_graph = True without doing things just so, you are
likely to get an inplace-modification error.

Probably the simplest way to address this issue with training GANs is
to rebuild the discriminator computation graph by calling – at added
computational cost – discriminator_fake_out = discriminator (...)
twice, once for the discriminator_fake_loss.backward()
backpropagation and then again for the generator_loss.backward()
backpropagation.

Problem is from this line generator_loss.backward().

Yes, as explained in the in-line comments I added to your code.

Also when I remove the line discriminator_optimizer.step() the problem disappears.

I don’t believe this. For the version of the code you posted, you will get the
“backward a second time” error after calling generator_loss.backward(),
regardless of whether discriminator_optimizer.step() was called or not.

Perhaps in a different version of your code you had an inplace-modification
error that removing discriminator_optimizer.step() appeared to fix.

If you have (or do) come across inplace-modification errors, it will be
because discriminator_optimizer.step() is modifying discriminator’s
parameters inplace.

A discussion about fixing inplace-modification errors that includes a
toy-GAN example can be found in this post:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

Best.

K. Frank