Concatenate two tensor for a linear layer

nlp
1

So i have one output from a linear layer dimension torch.randn(8, 512, 2048).
i want to concatenate it with another tensor torch.randn(8, 512) to a
linear layer nn.Linear(2048+512, num_classes). how to do it?
thx in advance

2

You won’t be able to directly concatenate these tensors as the second one misses a dimension.
You could unsqueeze the missing dim1 and expand the tensor assuming you want to repeat its values 512 times.

1 Like
3

i have that for every sample input( batchsize = 8)( tensor torch.randn(8, 512)) i have a tensor of 512 which add information and should improve the model prediction. how to integrate this information, i thnk it doesnt make sense to repeat the values

4

This would work:

a = torch.randn(8, 512, 2048)
b = torch.randn(8, 512)

b = b.unsqueeze(1).expand(-1, 512, -1)

c = torch.cat((a, b), dim=2)
print(c.shape)
# torch.Size([8, 512, 2560])