Dealing with in-place operation in custo function

autograd
1

Hello, I am trying to implement the function below and compute its gradient using backward(), but I have an in-place operation that I cannot solve, below I show an option I tried and didn’t work.

def modified_gram_schmidt(A):
    m, n = A.shape

    Q = torch.zeros((m, n), requires_grad=True).clone()
    R = torch.zeros((n, n), requires_grad=True).clone()

    for j in range(n):
        v = A[:, j].clone()
        for i in range(j):
            R[i, j] = torch.dot(Q[:, i], v)
            v_prev = v.clone()
            v = v_prev - torch.dot(Q[:, i], v_prev) * Q[:, i]
        R[j, j] = torch.norm(v)
        v = v / torch.norm(v)
        Q[:, j] = v.clone()

    return Q, R

The operation which is causing the issue is v = v_prev - torch.dot(Q[:, i], v_prev) * Q[:, i], when I try to compute the gradient by Q.sum().backward() I have the error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation. I would appreciate any help.

2

Hi,

A simple solution is just to remove the inplace change into Q and only create it at the end:

def modified_gram_schmidt(A):
    m, n = A.shape

    Q = []
    R = torch.zeros((n, n), requires_grad=True).clone()

    for j in range(n):
        v = A[:, j].clone()
        for i in range(j):
            R[i, j] = torch.dot(Q[i], v)
            v_prev = v.clone()
            v = v_prev - torch.dot(Q[i], v_prev) * Q[i]
        R[j, j] = torch.norm(v)
        v = v / torch.norm(v)
        Q.append(v)

    return torch.stack(Q), R
3

Thank you so much for the quick reply, that solved the problem!