Understand mark_dirty()

yzhu · February 2, 2017, 3:30am

So I read the inline documentation about mark_dirty() here:

pytorch/pytorch/blob/fb2d28f477c76bd94e3e3e9d2f424caa295d75c3/torch/autograd/function.py#L69


    """
    self.to_save = tensors


def mark_dirty(self, *args):
    """Marks given tensors as modified in an in-place operation.


    **This should be called at most once, only from inside the**
    :func:`forward` **method, and all arguments should be inputs.**


    Every tensor that's been modified in-place in a call to :func:`forward`
    should be given to this function, to ensure correcness of our checks.
    It doesn't matter wheter the function is called before or after
    modification.
    """
    self.dirty_tensors = args


def mark_shared_storage(self, *pairs):
    """Marks that given pairs of distinct tensors are sharing storage.


    **This should be called at most once, only from inside the**
    :func:`forward` **method, and all arguments should be pairs of

I don’t quite understand what extra checks are needed for inplace operators. Would be great if the devs can give some hints. Thanks!

smth · February 2, 2017, 3:49am

If you are doing an in-place operation, and further operate on the original Tensor, the backward gradients might be wrong.

Let’s take a small example:

y = x^2 z = x^2.

In this case, the gradient is 2x.
So, the input is needed to compute the gradients in the backward.

If we do all the operations out-of-place, we can hold onto the value of x and it’s not a problem to compute correct gradients.

However, if we do the second operation in-place via: z = x.pow_(2), where x is a Variable, we cannot compute the backward pass of y = x^2 correctly.

on all Variables, we have an internal version counter to track these things, and mark_dirty ensures that this version counter is correctly calculated.
If the user does an operation where the backward cannot be correctly computed, then an error is thrown.

yzhu · February 2, 2017, 4:02am

Thanks a lot for the explanation, Soumith