Torch.cat and backpropagation

vision
1

Does torch.cat work with backpropagation?

2

Hi,

Yes,
All operations that works with floating point numbers work with backprop in pytorch (or you will get an error when trying).

3

Thank you!
So if I have something like that
class NeuralNet2(nn.Module):
def init(self):
super(NeuralNet2, self).init()

    self.input = nn.Linear(2, 40)
    self.hiddenLeft = nn.Linear(40, 2)
    self.hiddenRight = nn.Linear(40, 2)
    self.out = nn.Linear(4, 6)

def forward(self, x):

    x = self.input(x)
    xLeft, xRight = torch.sigmoid(self.hiddenLeft(x)), torch.sigmoid(self.hiddenRight(x))
    x = torch.cat((xLeft, xRight), 2)
    x = F.sigmoid(self.out(x))

    return x

Is it necessary to re-implement the backward? Or how would it work in this case?

4

No you don’t need to reimplement the backward. Since we can just backprop through the cat operation, gradients will be computed for all your parameters.

1 Like
5

So the concatenation causes something similar to what would be a sequential model?

6

Not sequential because your modules don’t operate on the result of the previous one.
It just concatenates two Tensors. That is it.

7

That is to say, independently of the concatenation, the neurons will be able to keep a history of who precedes them at the time of updating the weights

8

the neurons will be able to keep a history of who precedes them at the time of updating the weights

Not sure what that means.
But if the question is “Will it be able to compute gradients?”, then the answer is yes.

9

If you would like to access specific positions in a tensor, how would you do it most efficiently?
tensor([[[0.1743, 0.2439, 0.2543]],
[[0.1325, 0.0778, 0.2292]],
[[0.3474, 0.1554, 0.1000]],
[[0.1599, 0.0283, 0.0305]],
[[0.3842, 0.3997, 0.2972]],
[[0.1876, 0.0815, 0.3047]],
[[0.2280, 0.1096, 0.2919]],
[[0.3843, 0.3552, 0.1672]]])

And I want the tensor but only with the first and third columns???

10

Hi,

The simplest would be to use indexing: t[:, (0, 2)].

11

Thanks!
It would not be so t[:, :, (0, 2)]?

12

Ho yes, sorry, though it was a 2D Tensor.

13

And if I wanted to concatenate a list of tensors, how would I do it with cat?

14

cat takes any iterable as input. So you can just give the list to cat.

15

Is it proper to do this torch.optim.Adam(net.parameters()) after this net(inputs.float())? I need it so I define the network in the forward for necessary reasons.

    for i, [inputs, labels] in enumerate(trainloader):
        inputs = inputs.to(device)
        labels = labels.to(device)

        # Forward + backward + optimize
        outputs = net(inputs.float())

        optimizer = torch.optim.Adam(net.parameters())
        optimizer.zero_grad()

        loss = criterion(outputs, labels.float())
        loss.backward()
        optimizer.step()
16

I would advise against it because optimizers have states that you want to keep from one iteration to the next.
You can provide a dummy input to the network when you create it if you really need to know the input size to create all the Parameters.

17

I get the following error if I put the sentence before the forward:
ValueError: optimizer got an empty parameter list.

And it’s happening to me because I define the net in the forward, because the parameters of the first layers are defined at that time, what do you recommend?

18

Is it possible to put dummy values to create the layers? but in that case, this net.parameters() is not true at the time, how can this be updated later with the real parameters?

19

I think you’re not using Parameters properly.
They are supposed to contain a set of weights that you iteratively update so that they contain the right value. So you should not re-create them all the time as they are supposed to be updated by the optimizers iteratively.

20

So it is necessary to define the net, that is nn.Linear(n1, n2) before the forward? In my case I only have the values of n1 and n2 computed during the forward, what can I do?