Try moving all the optimizer steps to the very end after all the backwards have completed
See these two similar issues:
- MobileFSGAN - One of the variables needed for gradient computation has been modified by an inplace operation
- RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]] is at version 2; expected version 1 instead - #14 by soulitzer