RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 64, 3, 3, 3]] is at version 2; expected version 1 instead

I was trying to make 3D GAN with FMRI dataset with codes from 3dbraingen/VAEGAN_ADNI_train.ipynb at master · cyclomon/3dbraingen · GitHub

And it keeps making errors that I don’t understand…

can anybody help me to solve this problem?


Have you tried enabling anomaly mode as suggested in the error message?

yes i had!

But it keeps making errors :frowning:

It is not supposed to remove the error but give you more information about where it comes from.

hmm i think tried .mean() → torch.mean() and inplace to True for nn.ReLU and Leaky …

any other suggestions?

Well, it is hard to say without the second stack trace that the anomaly mode will give.
You should enable anomaly mode as suggested and share the second stack trace here.

I applied the anomaly mode ‘True’

should I put “torch.autograd.set_detect_anomaly(True)” in training steps?


Given the stack, it looks like the issue is that the input/output to the conv layer it points to has been modified inplace.
Do you use inplace versions of activations in your code

When I used the model that makes no error(Alpha_GAN), it keeps making it error.

So I am not really sure that its a conv layers’ problem

The model that I am currently using looks like this.

The training cod is here.