Why does autograd track things like `torch.cat`?

I’m coding a timeseries autoencoder that reconstructs the elements of the input reverse order, one at a time. I’m using the torch.cat operation to concatenate each element back into sequence form.

        # reconstruct remaining elements
        for i in range(1, self.seq_len):
            hs_i, cs_i = self.cell(x_i, (hs_i, cs_i))
            x_i = self.dense(hs_i)
            x = torch.cat([x, x_i])
        return x

I noticed that the output has grad_fn=<CatBackward>.

I was wondering why the autograd system would track things like collecting elements into a sequence? Will this mess with my backprop when it’s time to train?

torch.cat will be tracked by Autograd and the backward pass will pass through this operation into the incoming tensors. This is useful in case you want to e.g. concatenate outputs of two different models, calculate the loss, and compute the gradients for both models.
If you don’t want to track this operation, you could .detach() the input tensors.

1 Like

Okay, that makes sense! So in my case I’ve got an LSTM cell in a for-loop and I’m concatenating the results of each pass into my final output sequence… back prop won’t go through that tensor unless the concatenation is tracked… is that a correct summary of what you are saying?

Yes and by default the torch.cat operation is tracked, so based on your description you would append the outputs of the model with the complete computation graph and would then backpropagate through all these outputs and the model.

1 Like

I understand that torch.cat is tracked by Autograd and that the backward pass will “pass through” these operations into the incoming tensors. Is there a way using the “loss.grad_fn.next_functions” commands to reach the leave nodes to verify this?