Repeat examples along batch dimension

Hi, I’m trying to repeat tensors along the batch dimension.

Ex)
We have a batch (8 x 3 x 224 x 224) where its size is 8
and let’s say it is called as [a, b, c, d, e, f, g, h], and each alphabet denotes an example in the batch.

Then, I want to repeat each of them three times, resulting in EXACTLY FOLLOWING ORDER [a, a, a, b, b, b, c, c, c, d, d, d, …, h, h, h]. Same elements should come in succession.
Thus, the final dimension is (24 x 3 x 224 x 224).

How can I make it possible?

3 Likes
new_tensor = torch.cat(3*[orig_tensor])

would probably do the job.

7 Likes

An alternative way is to use torch.repeat(). So with torch.repeat(), you can specify the number of repeats for each dimension:

>>> a = torch.randn(8, 3, 224, 224)
>>> b = a.repeat(3, 1, 1, 1)
>>> b.shape
torch.Size([24, 3, 224, 224])
28 Likes

Sorry for the confusion. I omit to mention that the same element should be in succession.

1 Like

Oh, in that case, neither of these solutions work:

>>> t = torch.tensor([[1, 2, 3], [4, 4, 4]])                                                        
>>> t
tensor([[1, 2, 3],
        [4, 4, 4]])
>>> torch.cat(3*[t])
tensor([[1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4]])
>>> t.repeat(3, 1)
tensor([[1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4]])

But based on the answer from How to tile a tensor? we can do that so that the repeated elements are succession:

def tile(a, dim, n_tile):
    init_dim = a.size(dim)
    repeat_idx = [1] * a.dim()
    repeat_idx[dim] = n_tile
    a = a.repeat(*(repeat_idx))
    order_index = torch.LongTensor(np.concatenate([init_dim * np.arange(n_tile) + i for i in range(init_dim)]))
    return torch.index_select(a, dim, order_index)

then, using this function, the repeated elements will be as follows:

tile(t, 0, 3)
tensor([[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3],
        [4, 4, 4],
        [4, 4, 4],
        [4, 4, 4]])
7 Likes

Great! Thanks a lot.

Borrowing from my answer, for anyone new looking for this issue, an updated function has also been introduced in pytorch - torch.repeat_interleave() to address this issue in a single operation.

So for t = torch.tensor([[1, 2, 3], [4, 4, 4]]) one can use torch.repeat_interleave(t, repeats=3, dim=0) to obtain:

tensor([[1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.],
        [4., 4., 4.],
        [4., 4., 4.],
        [4., 4., 4.]])
30 Likes

Suppose a tensor is of dimension (9,10), say it A, A.repeat(1,1) would produce same tensor as A.
Calling A.repeat(1,1,10) produces tensor of dimension 1,9,100
Again calling A.repeat(1,2,1) produces 1,18,10.
It look likes that from right to left, element wise multiplication is happening from the input of repeat

Einops recently got support for repeat-like patterns. Examples:

# np.repeat behavior (copies are in succession, like aaabbbcccddd as described by topic starter)
einops.repeat(x, 'b c h w -> (b copy) c h w', copy=3)
# np.tile behavior (whole sequence is repeated 3 times like abcdabcdabcd)
einops.repeat(x, 'b c h w -> (copy b) c h w', copy=3)

You can repeat/tile multiple axes independently within one operation.

4 Likes

man, really helpful. I’d like to say a great thanks to you

1 Like