Cant perform forward pass twice in DDP

distributed
1

I am trying to forward pass 2 different inputs with the same model as shown below:

for epoch in range(num_epochs):
        dataloader.sampler.set_epoch(epoch)
        for batch_index, (real,_) in enumerate(dataloader):
            
            disc.zero_grad()
            
            real=real.to(rank)
            noise=torch.randn((batch_size,z_dim,1,1)).to(rank)
            fake_img=gen(noise)
            fake_img_clone=fake_img.detach().clone()
            disc_real=disc(real).reshape(-1)
            lossD_real=critereon(disc_real,torch.ones_like(disc_real))
            
            disc_fake=disc(fake_img.detach()).reshape(-1)
            lossD_fake=critereon(disc_fake,torch.zeros_like(disc_fake))
            lossD = (lossD_fake+lossD_real)/2
                        
            opt_disc.step()

However, I keep getting the error “one of the variables needed for gradient computation has been modified by an inplace operation”
Setting torch.autograd.set_detect_anomaly(True, check_nan=True) shows that the error occurs in disc_real=disc(real).reshape(-1), but when I manually debug it, the error occurs only when I add the second forward pass line disc_fake=disc(fake_img.detach()).reshape(-1)

I am currently using the latest version of pytorch. Please help me solve this :frowning:

2

You are never calling backward in the posted code snippet so I guess you removed important parts of the code. Could you revisit it, please?