How detach() affect to the model?

[oasjd7]

Network flow:
[Model_1] -> [Model_2] -> [Model_3]

optimizer1 = {Model_1 + Model_2}
optimizer2 = {Model_3}

When I train with optimizer2,
If I don’t use detach() to input of model_3, this can affect to Model_1 and Model2 ???
Model_3(input.detach()) or Model_3(input)

[JuanFMontesinos (Juan Montesinos)]

it depends on were input comes from. If input is the output of another model it will break backprop . If it’s just a tensor nothing happens.

[Eta_C]

Copied from this topic.

@ptrblck

detach() operates on a tensor and returns the same tensor, which will be detached from the computation graph at this point, so that the backward pass will stop at this point.
detach_() is the inplace operation of detach() .

@ptrblck

They might yield the same result, but in specific edge cases you might get different and unexpected results:

• An optimizer, which doesn’t hold certain parameters, won’t update them. However other optimizers could still update these parameters, if they are passed.
• If you are using weight decay, have updated some parameters in the past and are now freezing them, these parameters could still be updated, if they were passed to the optimizer.
• detach() could act in the same way as the second point.

That being said, I would prefer the explicit way instead of relying implicitly that no side effects can occur.