Concatenate two tensor for each timestep

Hello all,
I have a problem concatenating two features tensor when coding an encoder-decoder network in Pytorch. Specifically, in forward function of my network, I have tensor x of size batch x feature1, the result of GRU. And tensor y of size seqLen x batch x feature2. I want to concatenate x into each timestep of y, should output z of size seqLen x batch x (feature1 + feature2).
I have tried allocating a Variable and iterate each timestep, but it doesn’t work. My code looks like this:

def forward(...):
        ...
        z = Variable(torch.zeros(y.size(0), y.size(1), 1024))
        for i in range(z.size(0)):
            z[i] = torch.cat((y[i], x), 1)

        z = self.decoderGRU(z, init_decoder_GRU)
        decoded = self.classLinear(z.view(z.size(0)*z.size(1), z.size(2))) #error here.
        ...

Error message:

AttributeError: 'tuple' object has no attribute 'view'

What is the best way to do this?

2 Likes

Just to be sure, you want to replicate x along the first (seqLen) dimension when concatenating it, right?

If so, this should do the trick for you and will be much more efficient:

# Replace the for loop with this line. It will expand x to be of size seqLen x batch x feature1
# and concatenate it with y in one go. 
z = torch.cat([y, x.unsqueeze(0).expand(seqLen, *x.size())], 2)

About the error message: it seems that the error is related to something else - GRU returns two outputs - the output and hidden state (see the docs), so if you want to retrieve only the output you have to do this:

z = self.decoderGRU(z, init_decoder_GRU)[0]
1 Like

Thank you. Both solve my problem. Never thought of using expand that way.

Hi, I wonder if it’s convenient for you to share your encoder-decoder code with me, as I am trying to implement a similar network?

Hi,

I have a similar problem. I have two tensors A and B and both batch, len, features and I want to concatenate every row A with every row of B.

Something like [wi; wj]