Gradient computation issue due to inplace operation, unsure how to debug for custom model

[KFrank]

Hi Sahil!

@Zahl

I have yet to properly understand when PyTorch is ok with an inplace operations.

An inplace modification of a tensor that is in the computation graph does
not necessarily cause an error – it depends on whether the that tensor’s
value is actually used in the backward pass. For some explanation, please
see this post:

When inplace operation are allowed and when not

Hi Jie (and Srishti)! The part that can be confusing is that a given tensor used in the forward pass may or may not be used in the backward pass. It depends on the math of the gradient computation (and the details of how the relevant backward() function is implemented). It’s not always obvious. Here are some explanatory examples: The derivative of exp (x) is exp (x). So it makes sense that pytorch saves the output of exp (x) when it is computed during the forward pass so that it can …

In fact for this code example I was actually able to find a fix by replacing

out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]

with

out[k] = alphas_p @ inter[k].clone()

The cloning being the key fix.

From your first post:

@Zahl

          inter[k, i] = skills[i](x[k].clone())
        out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]

inter is also in the computation graph and is also being modified inplace
(assignment to indexed elements). Because cloning inter fixes your inplace
error, modifying inter via indexing was likely the cause of your error.

Best.

K. Frank