Model does not change during training

1

Hi I implemented a Gan network. the discriminator does not change during training… loss is BCEloss and the last layer has a sigmoid activation function…
list(discriminator.parameters())[0].grad is all zeros…

import torch.nn.functional as F
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator,self).__init__()
        self.conv1= nn.Sequential(
            nn.Conv2d(2,128, kernel_size=(4,4), stride=2,padding=5),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2)
        )#32
        self.conv2= nn.Sequential(
            nn.Conv2d(128,256,kernel_size=(4,4), stride=2,padding=5),
            nn.BatchNorm2d(256),
            nn.LeakyReLU(0.2)
        )#16
        self.conv3= nn.Sequential(
            nn.Conv2d(256,1, kernel_size=(4,4), stride=2,padding=5),
            nn.Sigmoid()
        )#8
    def forward(self,kr):
        
        out_conv1=self.conv1(kr)
        out_conv2=self.conv2(out_conv1)
        out_conv3=self.conv3(out_conv2)        
        return out_conv3

inputsize=(batch,2,32,32)
outputsize=(batch,1,8,8)
loss=nn.BCELoss()

2

Can you show your training code.

3

thanks for reply @Dwight_Foster ! Ofcourse…

generator = Generator()
discriminator = Discriminator()
loss_discriminator=nn.BCELoss()
discriminator_optimizer = torch.optim.Adam(discriminator.parameters(), lr=0.0002, betas=(0.5,0.999))
generator_optimizer = torch.optim.Adam(generator.parameters(), lr=0.0002, betas=(0.5,0.999))
for iteration,(x,r) in enumerate(Train_Loader):
        X_real,y_real=generate_real_samples(x.to(device))
        # 1. Train D on Real+Fake data
        discriminator.train()
        generator.eval()
        X_fake=generator(r.to(device))
        discriminator_optimizer.zero_grad()
        # 1.A Train D on Real data
        out_diss=discriminator(X_real)
        d_loss1=loss_discriminator(out_diss,y_real)
        d_loss1.backward()
        discriminator_optimizer.step()
        # 1.B Train D on Fake data
        d_loss2=loss_discriminator(discriminator(torch.cat([x.to(device),X_fake], dim=1)),torch.zeros(x.shape).to(device))
        d_loss2.backward()
        discriminator_optimizer.step()
        # 2. Train D on Real+Fake data
        #generator_optimizer.zero_grad()
        #discriminator.eval()
        #generator.train()
        #X_fake=generator(r.to(device))
        #loss_g=gen_loss(x.to(device),X_fake,discriminator(torch.cat([x.to(device),X_fake], dim=1)))
        #loss_g.backward()
        #generator_optimizer.step()
4

One thing you could try is add d_loss1 and d_loss2 together and then use that loss for the backward step. Also it doesn’t seem like you run it for multiple epochs. You should do that if you want it to improve. If you do use epochs can you show the full code for that thanks.

5

@Dwight_Foster thanks again… I added d_loss1 to d_loss2 as you said but it did not make any changes… yes I run it for multiple epochs… just a simple ‘for i in rage(100):’ at fist

6

Is the code you provided above in the epoch for loop or is some of it outside it because if you keep defining a new model every epoch that wouldn’t work.

7

I defined the model outside the for loop… I double checked the code… discriminator loss decreases during the training but when I print the weights they are constant… How this can happen?

8

That is interesting. For this training loop are you using the generator. It could be that the generator is just getting worse so the discriminator is getting better without changing. It’s good that your loss is decreasing though. If your model trains than you shouldn’t have to worry about the weights not changing when you print them out. Can you just show how you are printing them out?