Best way to concatenate these tensor dimensions

1

Hello, after reading this post (For beginners: Do not use view() or reshape() to swap dimensions of tensors!) regarding the usage of .view() and .reshape(), a doubt has come to my mind.

If I had a tensor ‘A’ of dimensions A=(N, H), being ‘N’ the batch dimensions and ‘H’ the dimensions of a LSTM, and a second tensor ‘B’, of dimensions B=(T) that represents a vector of time event of duration ‘T’, what would be the best way to concatenate both of them to obtain a vector ‘C’ of dimensions C=(N,H,T)?

Thanks!

2

You can’t concatenate them in that way.
See tensor ‘A’ contains H numbers N times, tensor ‘B’ only T numbers. How do you want to concatenate first numbers with others?

The only way to concatenate them is to replace all ‘H’ numbers with ‘T’ numbers but ‘H’ times. However in most cases it is a bad idea. You can do it by:

import torch

A = torch.rand(3, 2)
B = torch.rand(5)
C = B.repeat(A.shape[0], A.shape[1], 1)

the output will be:

A torch.Size([3, 2])
tensor([[0.1305, 0.8649],
        [0.2945, 0.3027],
        [0.6107, 0.6925]])
---
B torch.Size([5])
tensor([0.3653, 0.6501, 0.8864, 0.0936, 0.6012])
---
C torch.Size([3, 2, 5])
tensor([[[0.3653, 0.6501, 0.8864, 0.0936, 0.6012],
         [0.3653, 0.6501, 0.8864, 0.0936, 0.6012]],

        [[0.3653, 0.6501, 0.8864, 0.0936, 0.6012],
         [0.3653, 0.6501, 0.8864, 0.0936, 0.6012]],

        [[0.3653, 0.6501, 0.8864, 0.0936, 0.6012],
         [0.3653, 0.6501, 0.8864, 0.0936, 0.6012]]])
1 Like
3

Thanks for your answer!

Could it be done more elegantly through a NN? I mean, having two inputs A=(N, H) and B=(T), and desiring to have a third tensor C=(N,H,T) as an output, which could be interpreted as a different vector of dimensions ‘T’ for each of the elements of ‘A’, could I concatenate ‘A’ and ‘B’ though a nn.Linear?

Thanks again!

4

Yes, it could be done. However to train NN, you need examples of correct outputs . Sometimes some additional hidden layers are required, not only the single dense layer.

Maybe a better idea is describing the main problem as a whole. Maybe your approach is not well fitted to your original problem. It is possible that we will find a better solution.

1 Like
5

Thanks again, I will try to briefly summarise the problem and why I need to do this: I’m implementing a small module which is intended to operate as an attention network inside a bigger model.

This attention module has two different inputs: a vector that would be ‘B’ of dimension (F) and a matrix ‘A’, with dimensions (B, E). Combining both inputs, I want to obtain a third tensor ‘C’ of dimensions (B,E,F), which would be the ‘alpha’ of the attention module. That means that this variable ‘C’ multiply other variable ‘D’ of dimensions D=(B,E,F), obtaining the output [also of dimensions (B,E,F)].

Summarised, I need:

C = f(A,B).
C = softmax( C ), to make each element between (0,1).
output = C x D, where ‘x’ would be a dot product attention.

In all this scheme, I’m having troubles finding the optimal way to combine A=(B,E) and B=(F) into a 3D tensor of dimensions (B,E,X) where X could be change into the desired dimensions F with a nn.Linear.

Thanks again!

6

You can use this:


A = torch.rand(30, 20)
B = torch.rand(10)

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        
        self.linear = torch.nn.Linear(11, 10) #11 -> 1 from x + 10 from y
        self.softmax = torch.nn.Softmax(dim=1)
        
    def forward(self, x, y):
        x = x.unsqueeze(2)
        y = y.repeat(x.shape[0], x.shape[1], 1)
       
        z = torch.cat((x, y), dim = 2)
        z = self.linear(z)
        z = self.softmax(z)
        
        return z
    
net = Network()
C = net(A, B)
print(C.shape)

Of course you can design the model with additional layers.

1 Like