What is `in-place operation`?

15

Made a summary here. Hope it is helpful:

Best practice: Avoid inplace operations if it is not necessary as it changes the state of tensors silently. Non-inplace operations will make a copy before doing the operation. Thus, if an operation is inplace within a function, it affects the tensor’s state outside of the function while the non-inplace operation does not change the state unless you reassign it outside of the function.

e.g.

def inplace_op(X):
	X += 1
	return X

X = torch.rand(4, 2)
inplace_op(X) # X is changed without re-asigned to X

Summary of inplace operations:

  • x *= 3
  • X[…] = …
  • X.add_(1)

Common examples of inplace operations:

  • x += 1, x *= 3, …
  • x[2] = 2, X[0, 0] = 2
  • x[:, 3] = 3
  • X[2] /= 5
  • X[:, 3] /= 4
  • X[:, 3] = X[:, 3] / 4

These are not inplace operations:

  • x = x + 1
  • y = x.clone; y[0] += 100
4 Likes