Multiple Loss Functions in a Model

[InnovArul (Arul)]

I see. Even in this case, the final lossD.backward() faces this variable modified in-place scenario.

From @albanD 's answer here:

How to free graph manually?

Hi, Whenever the output Variable will go out of scope in python, the whole graph will be deleted. By default, some intermediary buffers are freed even before that to reduce peak memory usage (this is what is disabled when using retain_graph=True). But the graph and all intermediary buffers are only kept alive as long as they are accessible from python (usually from the output Variable), so running the last backward with retain_graph=True will only keep the intermediary buffers alive until they…

You can use del lossD instead of final lossD.backward() (to release computational graph). Can you try that?

Edit: Can you pack encoder and decoder into one optimizer (or) backward them together, if possible? The Encoder grad calculation is dependent on Decoder parameters as well. So, you can’t optimizer_decoder.step() before loss_ele.backward(). One solution is as follows:

calculate encoder loss, decoder loss, discriminator loss

# discriminator update
optimizer_disc.zero_grad()
lossD.backward(retain_graph=True)
optimizer_disc.step()

# encoder and decoder update
optimizer_encoder.zero_grad()
optimizer_decoder.zero_grad()

loss_generator = loss_encoder + loss_decoder
loss_generator.backward()

optimizer_decoder.step()
optimizer_encoder.step()

# to release the computation graph of the discriminator
del lossD