Pytorch tensor dimension wise concatenation

I need to achieve my targets by developing computationally efficient code. Here is the scenario:

x     # torch.Size([2, 5, 256]) (Batch, input_1st_dim, input_2nd_dim)

Now I want to concatenate the input_1st_dim index-wise, like 1st with all the following four, then 2nd with the next three, then 3rd with the next two, then 4th with the last one.
Finally, I want to get x as [2, 15, not sure about this dimension]

I can do it with nested loops and some extra lines of code. Here is my code:

   def my_fun(self, x):
        iter = x.shape[0]
        counter = 0
        new_x = torch.zeros((10, x.shape[1]), dtype=torch.float32, device=torch.device('cuda'))
        for i in range(0, x.shape[0] - 1):
            iter -= 1
            for j in range(0, iter):
                mean = (x[i, :] + x[i+j, :])/2
                new_x[counter, :] = torch.unsqueeze(mean, 0)
                counter += 1
        final_T = torch.cat((x, new_x), dim=0)
        return final_T
ref = torch.zeros((x.shape[0], 15, x.shape[2]), dtype=torch.float32, device=torch.device('cuda'))
for i in range (x.shape[0]):
    ref[i, :, :] = self.my_fun(x[i, :, :])

But, is there any computationally efficient way to code it?

Could you post this slow approach so that we could use it as a reference to compare potentially faster approaches, please?

I stuck with one error:
x.shape # torch.Size([5, 256])
new_x.shape # torch.Size([10, 256])
Now I want concatenate these two tensors row wise means in result want to get (15, 256) and for that I am using final_T = torch.cat((x, new_x), -1, device='cuda'), but getting error:

TypeError: cat() received an invalid combination of arguments - got (tuple, int, device=str), but expected one of:

  • (tuple of Tensors tensors, int dim, *, Tensor out)
  • (tuple of Tensors tensors, name dim, *, Tensor out)`

This should work:

x = torch.randn(5, 256)
new_x = torch.randn(10, 256)
out = torch.cat((x, new_x), dim=0)
print(out.shape)
# torch.Size([15, 256])
1 Like

your examples work, but in my example case its does not work. I tried final_T = torch.cat((x, new_x), dim=0, device='cuda')

Detailed Code is:
iter = x.shape[0]
counter = 0
new_x = torch.zeros((10, x.shape[1]), dtype=torch.float32)
for i in range(0, x.shape[0] - 1):
iter -= 1
for j in range(0, iter):
mean = (x[i, :] + x[i+j, :])/2
new_x[counter, :] = torch.unsqueeze(mean, 0)
counter += 1
final_T = torch.cat((x, new_x), dim=0)

Remove the device in torch.cat argument as it’s unexpected.

Thanks for efficiently responding. But after removing device, I got:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument tensors in method wrapper___cat)

In that case move the CPUTensor to the device via tensor.to(device), but don’t pass it as an argument to torch.cat.

1 Like

Thank you so much. Now its work.

I finished my version and updated the post with my code. Kindly check whether there is a more efficient way to accomplish this. Furthermore, please double-check that my logic matches the problem stated in the post. Thank you very much.

@ptrblck Waiting for your response. Thanks