Backward error: an in-place operation took place and prevents backprop

1

I’m tracing back the error and it points at this:

def zlogit(re, im):
    abs_ = t.sqrt(re**2 + im**2)
    ang = t.atan2(im, re)
    mask = ~(ang > 0)
    abs_[mask] = -1 * abs_[mask]
    return abs_

Where is the inplace operation that I did?

2

I solved it! Woah! The error was misleading and was pointing to

abs_ = t.sqrt(re**2 + im**2)
While in reality, the error is in:

abs_[mask] = -1 * abs_[mask]

That was the inplace opeartion. I had to create a clone of that tensor and modify the clone like this:

clone = abs_.clone()
clone[mask] = -1 * abs_[mask]

Is there a more efficient way? cause abs_ is wasted in my solution. Can the clone be a detached tensor and I go like abs_ = Dtached[mask] ? IDK.

3

No, sqrt’s backward wants abs_ to calculate d sqrt(x) = -0.5 / sqrt(x) .
If you absolutely needed the memory, you use checkpointing for zlogit, but most of the time that type of memory optimization is not worth the effort.

Best regards

Thomas

1 Like