# Torch.cat and backpropagation

**URL:** <https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802>\
**Category:** vision\
**Created:** [March 30, 2020, 2:21pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802 "2020-03-30T14:21:55Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 2:21pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/1 "2020-03-30T14:21:55Z")

</div>

Does torch.cat work with backpropagation?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 2:38pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/2 "2020-03-30T14:38:38Z")

</div>

Hi,

Yes,  
All operations that works with floating point numbers work with backprop in pytorch (or you will get an error when trying).

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 2:42pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/3 "2020-03-30T14:42:42Z")

</div>

Thank you!  
So if I have something like that  
class NeuralNet2(nn.Module):  
def **init** (self):  
super(NeuralNet2, self). **init** ()

```
    self.input = nn.Linear(2, 40)
    self.hiddenLeft = nn.Linear(40, 2)
    self.hiddenRight = nn.Linear(40, 2)
    self.out = nn.Linear(4, 6)

def forward(self, x):

    x = self.input(x)
    xLeft, xRight = torch.sigmoid(self.hiddenLeft(x)), torch.sigmoid(self.hiddenRight(x))
    x = torch.cat((xLeft, xRight), 2)
    x = F.sigmoid(self.out(x))

    return x

```

Is it necessary to re-implement the backward? Or how would it work in this case?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 3:24pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/4 "2020-03-30T15:24:45Z")

</div>

No you don’t need to reimplement the backward. Since we can just backprop through the cat operation, gradients will be computed for all your parameters.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 4:23pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/5 "2020-03-30T16:23:42Z")

</div>

So the concatenation causes something similar to what would be a sequential model?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 4:24pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/6 "2020-03-30T16:24:57Z")

</div>

Not sequential because your modules don’t operate on the result of the previous one.  
It just concatenates two Tensors. That is it.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 4:31pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/7 "2020-03-30T16:31:58Z")

</div>

That is to say, independently of the concatenation, the neurons will be able to keep a history of who precedes them at the time of updating the weights

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 4:39pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/8 "2020-03-30T16:39:02Z")

</div>

> the neurons will be able to keep a history of who precedes them at the time of updating the weights

Not sure what that means.  
But if the question is “Will it be able to compute gradients?”, then the answer is yes.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 5:32pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/9 "2020-03-30T17:32:11Z")

</div>

If you would like to access specific positions in a tensor, how would you do it most efficiently?  
tensor([[[0.1743, 0.2439, 0.2543]],  
[[0.1325, 0.0778, 0.2292]],  
[[0.3474, 0.1554, 0.1000]],  
[[0.1599, 0.0283, 0.0305]],  
[[0.3842, 0.3997, 0.2972]],  
[[0.1876, 0.0815, 0.3047]],  
[[0.2280, 0.1096, 0.2919]],  
[[0.3843, 0.3552, 0.1672]]])

And I want the tensor but only with the first and third columns???

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 6:08pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/10 "2020-03-30T18:08:53Z")

</div>

Hi,

The simplest would be to use indexing: `t[:, (0, 2)]`.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 6:14pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/11 "2020-03-30T18:14:55Z")

</div>

Thanks!  
It would not be so t[:, :, (0, 2)]?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 6:16pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/12 "2020-03-30T18:16:43Z")

</div>

Ho yes, sorry, though it was a 2D Tensor.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 30, 2020, 6:41pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/13 "2020-03-30T18:41:27Z")

</div>

And if I wanted to concatenate a list of tensors, how would I do it with cat?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 30, 2020, 6:44pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/14 "2020-03-30T18:44:57Z")

</div>

cat takes any iterable as input. So you can just give the list to cat.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 31, 2020, 2:47pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/15 "2020-03-31T14:47:49Z")

</div>

Is it proper to do this torch.optim.Adam(net.parameters()) after this net(inputs.float())? I need it so I define the network in the forward for necessary reasons.

```
    for i, [inputs, labels] in enumerate(trainloader):
        inputs = inputs.to(device)
        labels = labels.to(device)

        # Forward + backward + optimize
        outputs = net(inputs.float())

        optimizer = torch.optim.Adam(net.parameters())
        optimizer.zero_grad()

        loss = criterion(outputs, labels.float())
        loss.backward()
        optimizer.step()
```

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 31, 2020, 2:53pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/16 "2020-03-31T14:53:39Z")

</div>

I would advise against it because optimizers have states that you want to keep from one iteration to the next.  
You can provide a dummy input to the network when you create it if you really need to know the input size to create all the Parameters.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 31, 2020, 3:02pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/17 "2020-03-31T15:02:56Z")

</div>

I get the following error if I put the sentence before the forward:  
ValueError: optimizer got an empty parameter list.

And it’s happening to me because I define the net in the forward, because the parameters of the first layers are defined at that time, what do you recommend?

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 31, 2020, 3:07pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/18 "2020-03-31T15:07:11Z")

</div>

Is it possible to put dummy values to create the layers? but in that case, this net.parameters() is not true at the time, how can this be updated later with the real parameters?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [March 31, 2020, 3:12pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/19 "2020-03-31T15:12:07Z")

</div>

I think you’re not using Parameters properly.  
They are supposed to contain a set of weights that you iteratively update so that they contain the right value. So you should not re-create them all the time as they are supposed to be updated by the optimizers iteratively.

---

<div class="post-metadata">

**Author:** ![uhmbg](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9e8a1a/32.png) [@uhmbg](https://discuss.pytorch.org/u/uhmbg)\
**Post date:** [March 31, 2020, 3:19pm UTC](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802/20 "2020-03-31T15:19:41Z")

</div>

So it is necessary to define the net, that is nn.Linear(n1, n2) before the forward? In my case I only have the values of n1 and n2 computed during the forward, what can I do?

[Next page](https://discuss.pytorch.org/t/torch-cat-and-backpropagation/74802.md?page=2)