A GAN model in PyTorch does not learn

Can anyone kindly take a look at my jupyter notebook and kindly let me know what part I did is erroneous? Thanks so much… Here is the unlearning model I spent a whole day working on: https://github.com/tlkahn/my-notebooks/blob/master/GAN-pytorch.ipynb

Just by skimming through your code, it seems you are freeing the discriminator:



class GAN(nn.Module):
    """GAN model"""
    def __init__(self, generator, discriminator):
        super().__init__()
        self.generator = generator
        self.discriminator = discriminator
        for param in self.discriminator.parameters():
            param.requires_grad = False

    def forward(self, x):
        gen_img = self.generator(x)
        return self.discriminator(gen_img)

Which will not train it. Is this on purpose or were you planning on unfreezing the parameters at some point?

Here are two GAN in pytorch that are pretty simple and easy to follow if they help you.

Here is how to set up two models GEN and DESCRIM and train them

# Set models for training
Disc.train()
Gen.train()
for epoch in range(num_epochs):
    # Each batch
    for batch_i, (real_images, _) in enumerate(train_loader):
        batch_size = real_images.size(0)
        
        ## Important rescaling step ## 
        # rescale input images from [0,1) to [-1, 1)
        real_images = real_images*2 - 1
        # Train Discriminator
        d_optimizer.zero_grad()
        # Train with real images
        D_real = Disc(real_images)
        d_real_loss = real_loss(D_real, smooth=True) # Use label smoothing
        
        # Next step train with fake images
        # Generate fake images
        z = np.random.uniform(-1, 1, size=(batch_size, z_size)) # Random Noise
        z = torch.from_numpy(z).float() # Convert to a flot
        fake_images = Gen(z) # Predict with Generator (DO NOT TRAIN GENERATOR HERE) train one at a time
        
        # Computer fake loss
        D_fake = Disc(fake_images)
        d_fake_loss = fake_loss(D_fake)
        # add up loss and back prop
        d_loss = d_real_loss + d_fake_loss
        d_loss.backward()
        d_optimizer.step()
        
        '''
        # Generator training
        '''
        g_optimizer.zero_grad()
        # Generator fake images and train
        z = np.random.uniform(-1, 1, size=(batch_size, z_size))
        z = torch.from_numpy(z).float()
        fake_images = Gen(z)
        
        # Compute the discriminator losses on fake images 
        # using flipped labels!
        D_fake = Disc(fake_images)
        g_loss = real_loss(D_fake) # use real loss to flip labels
        
        # perform backprop
        g_loss.backward()
        g_optimizer.step()
        
        # Print some loss stats
        if batch_i % print_every == 0:
            # print discriminator and generator loss
            print('Epoch [{:5d}/{:5d}] | d_loss: {:6.4f} | g_loss: {:6.4f}'.format(
                    epoch+1, num_epochs, d_loss.item(), g_loss.item()))
    
    ## AFTER EACH EPOCH##
    # append discriminator loss and generator loss
    losses.append((d_loss.item(), g_loss.item()))
    
    # generate and save sample, fake images
    Gen.eval() # eval mode for generating samples
    samples_z = Gen(fixed_z)
    samples.append(samples_z)
    Gen.train() # back to train mode
1 Like

Thanks a lot. I pulled those lines out of GAN model, and move them to later training phase and it seems to be working. :slight_smile:

for param in self.discriminator.parameters():
            param.requires_grad = False

Thanks. Very useful references :slight_smile:

Hi Everyone i am getting RuntimeError: element 11 of tensors does not require grad and does not have a grad_fn. when I am running a GAN Architecture for getting the Gaussian Model and I am using WGANLOSS as well.

How to fix the error so that i can get th required output for my model.

Double post from here.

Hi Ptrblck,

I hope you are well. Sorry, I need to run a 3D GAN. my inputs are gray scale patches in 3D and I want to create 3D patch as well.
Can I use any 2D GAN, and just convert any 2d to 3d?
Could you please suggest me any pytorch link to have DCGAN in 3D ?

Many thanks

I don’t know any recent 3D DCGAN implementations, but you could try to use the approach of Voxel DCGAN or 3DGAN, which are both a bot older by now.

If you are working with static volumetric shapes, you could use the depth dimension as the channel dimension in a standard 2D GAN, although I don’t know, how well this would work.
I think your suggestion makes sense and you could try to replace all nn.*2D layers with their 3D equivalent.

Many thanks .I will tell you the results

Hi Ptrblck,

Sorry, I am using 2D-DCGAN generator and convert it to the 3D. The error is : Given input size per channel: (1 x 1 x 1). Calculated output size per channel: (5 x 5 x -157). Output size is too small.
I expect that instead of -157 to see ((5x5x33) (13x13x21),(21x21x11), would you please help me to find out why this happen? I started by 101x1x1 and my target output is (21x21x11)

 class Generator(nn.Module):
        def __init__(self, nz):
        super(Generator, self).__init__()
        self.nz=nz
        self.main = nn.Sequential(
            nn.ConvTranspose3d(101,33,kernel_size=(5,5,5), stride=(2,2,2), padding=(0,0,86), bias=False),
            nn.BatchNorm3d(33),
            nn.ReLU(True),
            nn.ConvTranspose3d(33, 21,kernel_size=(5,5,5), stride=(2,2,2) , padding=(0,0,24), bias=False),
            nn.BatchNorm3d(21),
            nn.ReLU(True),
            nn.ConvTranspose3d(21, 11, kernel_size=(5,5,5), stride=(2,2,2), padding=(4,4,17), bias=False),
            nn.Tanh())
 
    def forward(self, input):
        return self.main(input)

#### call the generator   nz=101
netG = Generator(101).to(device)
if (device.type == 'cuda') and (ngpu > 1): netG = nn.DataParallel(netG, list(range(ngpu)))
netG.apply(weights_init)
noise = torch.randn(b_size,nz,1,1,1,device=device)
fake = Generator(noise)

Hi Ptrblck,

I want to use zero mean and unit normalization for the discriminator inputs (not rescalling between [-1 1]). in this case I should remove the Tanh from generator. which activation function u recommend instead of Tanh in generator in the last layer? or I can run without any activation function in the end?

Hi, if your output spans between 0 and 1, you could use nn.Sigmoid()
Without using an activation will make your output an arbitrary number not between -1,+1 or 0,1.

Hi Ptrblck,

I implement my conditional Gan. The batch size is 64 and input patch size of 21x21, and the condition which I pass is 168 different volumes number which can be from 1 to 100 for example =tensor([ 3, 36, 19, 12, 16, 6, 7, 2, 45, 12, 65, 44, 17, 8, 15, 15, 14, 47, 20, 9, 16, 25, 56, 11, 22, 8, 5, 3, 7, 6, 25, 10, 36, 1, 17, 2, 22, 3, 10, 13, 9, 14, 15, 11, 20, 16, 3, 10, 4, 18, 1, 15, 9, 6, 16, 55, 1, 14, 6, 17, 6, 6, 10, 7]).

  1. img_size=21
  2. N_Class=168 (168 different volumes as conditions)
    3)lr1=0.0002
    4)lr2=0.0002
    5)batch size =64
    6)optimized Adam as default
    7)ngf = 64
    8)criterion = nn.BCELoss()
    real_label = 1
    fake_label = 0

I applied this code. There is no error from my code and it runs, but it makes noise for me and fake images are not meaningful they are just noise.
Would you please help me with that? The code is

# custom weights initialization called on netG and netD
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)
        
# Generator Code

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        
        self.l1= nn.Sequential(nn.ConvTranspose2d(100, ngf * 4, 3, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 4), nn.ReLU(True))
        self.l2 = nn.Sequential(nn.ConvTranspose2d(168, ngf * 4, 3, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 4), nn.ReLU(True))

        self.l3=nn.Sequential(nn.ConvTranspose2d(ngf * 8, ngf *4 , 3, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 4),
        nn.ReLU(True))

        self.l4=nn.Sequential(nn.ConvTranspose2d( ngf * 4, ngf * 2, 3, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 2),
        nn.ReLU(True))
    
        self.l5=nn.Sequential(nn.ConvTranspose2d( ngf * 2, ngf, 3, 2, 1, bias=False),
        nn.BatchNorm2d(ngf),
        nn.ReLU(True))

        self.l6=nn.Sequential(nn.ConvTranspose2d( ngf, 1, 3, 2, 3, bias=False),nn.Sigmoid())

    def forward(self, input,Volume):
        x =self.l1(input)

        y = self.l2(Volume)

        xx = torch.cat([x, y], 1)
        output = self.l3(xx)
        output = self.l4(output)
        output = self.l5(output)
        output = self.l6(output)

        return output
        
    
# Create the generator
netG = Generator().to(device)
#  to mean=0, stdev=0.2.
netG.apply(weights_init)
#print(netG)
fixed_noise = torch.randn(64, nz, 1, 1, device=device)
fixed_noise_Se = torch.randn(4500, nz, 1, 1, device=device)

#-----------Discriminator ------------------

class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.l1= nn.Sequential(nn.Conv2d(1, int(ndf/2), 4, 2, 1, bias=False),nn.LeakyReLU(0.2, inplace=True))
        self.l2= nn.Sequential(nn.Conv2d(168, int(ndf/2), 4, 2, 1, bias=False),nn.LeakyReLU(0.2, inplace=True))
        self.l3=nn.Sequential(nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),nn.BatchNorm2d(ndf * 2),nn.LeakyReLU(0.2, inplace=True))
        self.drop_out3 = nn.Dropout(0.5)
        self.l4= nn.Sequential(nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False), nn.BatchNorm2d(ndf * 4), nn.LeakyReLU(0.2, inplace=True))
        self.drop_out4 = nn.Dropout(0.5)
        self.l5= nn.Sequential(nn.Conv2d(ndf * 4, 1, 4, 2, 1, bias=False),nn.Sigmoid())

    def forward(self, input,Volume):
        
         x =self.l1(input)

         y = self.l2(Volume)

         out = torch.cat([x, y], 1)

         out = self.l3(out)
         out=self.drop_out3(out)

         out=self.l4(out)

         out=self.drop_out4(out)

         out=self.l5(out)
      
         return out
    
# Create the Discriminator
netD = Discriminator(ngpu).to(device)
# Apply the weights_init function to randomly initialize all weights
netD.apply(weights_init)

# Setup Adam optimizers for both G and D
optimizerD = optim.Adam(netD.parameters(), lr=lr1, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr2, betas=(beta1, 0.999))


# Training Loop
print("Starting Training Loop...")
# For each epoch
for epoch in range(num_epochs):

    # For each batch in the dataloader
    for pos, neg in zip(trainloader,trainloaderNeg):
        images1,labels,Volumes=pos
        images1=images1.float()
        Volumes=Volumes.long()
        Negpach=neg
        Negpach=Negpach.float()
    
        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
# ----------------Format batch inputs ----------
        
        real_cpu = images1.to(device)
# ------- to add volumes as condition------------
        Real_volume=Volumes.to(device).long().squeeze(1)
        Real_volume=Real_volume.type(torch.LongTensor)

# ----------------label for Dis1 ------------
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        label=label.to(device)
        
# -------- pass the Volumes (condition) to the discriminator -------------
    
        real_y = torch.zeros(batch_size, N_Class)
        real_y = real_y.scatter_(1, Real_volume.view(batch_size, 1), 1).view(batch_size, N_Class, 1, 1).contiguous()
        real_y = Variable(real_y.expand(-1, -1, img_size, img_size)).to(device)
        netD=netD.float()
# --- apply dis on conditions --------------
        output = netD(real_cpu,real_y).view(-1)

        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        
        # Calculate gradients for D in backward pass
        errD_real.backward()
    
        ## Train with all-fake batch
# ------------Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        
# --------- Generate fake image batch with G and pass the condition ---------
        netG=netG.float()
        gen_labels = (torch.rand(batch_size, 1) * N_Class).type(torch.LongTensor)
        gen_y = torch.zeros(batch_size, N_Class)
        gen_y = Variable(gen_y.scatter_(1, gen_labels.view(batch_size, 1), 1).view(batch_size, N_Class,1,1)).to(device)
        fake = netG(noise,gen_y)
    
#------------- multiply with negative patch ---------------
      
        fake44=torch.mul(fake,Negpach)
         
# ---- labels for Dis for fake as input---------------     
        label.fill_(fake_label)
        label=label.to(device)
# --------- pass condition for dis---------
        gen_y_for_D = gen_y.view(batch_size, N_Class, 1, 1).contiguous().expand(-1, -1, img_size, img_size)
        output = netD(fake44.detach(),gen_y_for_D).view(-1)
        
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()
        
        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
# ---- labels for update G--------------

        label.fill_(real_label)  # fake labels are real for generator cost
#        label=torch.mul(label,0.9)    
        label=label.to(device)
# ----- apply dis on G with real label -----------

        output = netD(fake44.detach(),gen_y_for_D).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        # Update G
        optimizerG.step()
    
    
    if epoch % 1 == 0:     
         with torch.no_grad():
            fake = netG(fixed_noise,gen_y).detach().cpu()

            plt.close("all")
            plt.figure()
            plt.figure(figsize=(8,8))
            plt.axis("off")
            plt.title("Fake Images epoch "+str(epoch))
    
            plt.imshow(np.transpose(vutils.make_grid(fake.detach().to(device)[:64], padding=2, normalize=True,range=(0.2,1)).cpu(),(1,2,0)))
            plt.savefig(os.path.join(root_dirDurringTraining13+'/'+'Epoch='+str(epoch)+'Seed='+str(manualSeed))+'fakesor2.jpg')

            fake55=torch.mul(fake,Negpach)


            plt.close("all")
            plt.figure()
            plt.figure(figsize=(8,8))
            plt.axis("off")
            plt.title("Fake Images epoch "+str(epoch))
        
            plt.imshow(np.transpose(vutils.make_grid(fake55.detach().to(device)[:64], padding=2, normalize=True,range=(0,.5)).cpu(),(1,2,0)))
            plt.savefig(os.path.join(root_dirDurringTraining13+'/'+'Epoch='+str(epoch))+'fakesmul55.jpg')
            
    torch.save(netG.state_dict(), '%s/netG_epoch_%d.pth' % (root_dirDurringTraining15, epoch))
    torch.save(netD.state_dict(), '%s/netD_epoch_%d.pth' % (root_dirDurringTraining15, epoch))

Unfortunately, I cannot be really useful here, as I’m not a GAN expert. :confused:
Generally, I would recommend to take a look at similar architectures of conditional GANs and try to have a look at the tricks, which were used to make the training converge.

Hi Ptrblck,

I have a question from DCGAN.Would you please tell me what exactly happen in " errD = errD_real + errD_fake" and what is the relation to the optimizer, because in the previous lines the gradient was computed two times , how the OptimizedD understand to get (errD) to optimize?

netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()
```

The gradients will be calculated by

errD_real.backward()
errD_fake.backward()

not by errD, as .backward() is never used on it in the code snippet.
I guess errD is just used to print the sum of the real and fake loss for the discriminator.