Questions about overwriting variables in pytorch

1

I’m confused about overwriting values in pytorch and when you should or should not (link to docs)

I would like some general suggestions if anyone has any, but here are a few examples that I’m confused about:

loss += criterion(output, target) #is overwriting loss acceptable?

or

for _ in range(5):
   output = model(input)  #Am I overwriting the output?
   loss += criterion(output, target)
loss.backward()

In the example above, I overwrite the Variable output multiple times in the for loop.

  1. Am I “overwriting values required to compute gradients” with this in-place operation?
  2. Does code like this make the autograd computational graph slower in some way?
  3. What if I was using a Variable with thousands of elements instead of a scalar?

thank you

1 Like
2

  1. The line output = model(input) overwrites output in so far as it reassigns the name output to point to new data, in this case the return value of model(input). If some other code has kept a reference to the old data then python keeps the old data around. In this case the next line loss += criterion(output, target) uses the value of output to calculate the loss, and PyTorch updates the computation graph that it will use to calculate the gradients.

  2. This sort of code shouldn’t make things slower at all given that output is just a name that points to a location of data stored in memory.

  3. The size of the Variable is of no significance.

1 Like