Repeat examples along batch dimension

#1

Hi, I’m trying to repeat tensors along the batch dimension.

Ex)
We have a batch (8 x 3 x 224 x 224) where its size is 8
and let’s say it is called as [a, b, c, d, e, f, g, h], and each alphabet denotes an example in the batch.

Then, I want to repeat each of them three times, resulting in EXACTLY FOLLOWING ORDER [a, a, a, b, b, b, c, c, c, d, d, d, …, h, h, h]. Same elements should come in succession.
Thus, the final dimension is (24 x 3 x 224 x 224).

How can I make it possible?

3 Likes
#2 
new_tensor = torch.cat(3*[orig_tensor])

would probably do the job.

7 Likes
#3

An alternative way is to use torch.repeat(). So with torch.repeat(), you can specify the number of repeats for each dimension:

>>> a = torch.randn(8, 3, 224, 224)
>>> b = a.repeat(3, 1, 1, 1)
>>> b.shape
torch.Size([24, 3, 224, 224])
28 Likes
#4

Sorry for the confusion. I omit to mention that the same element should be in succession.

1 Like
#5

Oh, in that case, neither of these solutions work:

>>> t = torch.tensor([[1, 2, 3], [4, 4, 4]])                                                        
>>> t
tensor([[1, 2, 3],
        [4, 4, 4]])
>>> torch.cat(3*[t])
tensor([[1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4]])
>>> t.repeat(3, 1)
tensor([[1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4],
        [1, 2, 3],
        [4, 4, 4]])

But based on the answer from How to tile a tensor? we can do that so that the repeated elements are succession:

def tile(a, dim, n_tile):
    init_dim = a.size(dim)
    repeat_idx = [1] * a.dim()
    repeat_idx[dim] = n_tile
    a = a.repeat(*(repeat_idx))
    order_index = torch.LongTensor(np.concatenate([init_dim * np.arange(n_tile) + i for i in range(init_dim)]))
    return torch.index_select(a, dim, order_index)

then, using this function, the repeated elements will be as follows:

tile(t, 0, 3)
tensor([[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3],
        [4, 4, 4],
        [4, 4, 4],
        [4, 4, 4]])
7 Likes
#6

Great! Thanks a lot.

#7

Borrowing from my answer, for anyone new looking for this issue, an updated function has also been introduced in pytorch - torch.repeat_interleave() to address this issue in a single operation.

So for t = torch.tensor([[1, 2, 3], [4, 4, 4]]) one can use torch.repeat_interleave(t, repeats=3, dim=0) to obtain:

tensor([[1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.],
        [4., 4., 4.],
        [4., 4., 4.],
        [4., 4., 4.]])
30 Likes
Interpolate a tensor dimension from x to nx
#8

Suppose a tensor is of dimension (9,10), say it A, A.repeat(1,1) would produce same tensor as A.
Calling A.repeat(1,1,10) produces tensor of dimension 1,9,100
Again calling A.repeat(1,2,1) produces 1,18,10.
It look likes that from right to left, element wise multiplication is happening from the input of repeat

#9

Einops recently got support for repeat-like patterns. Examples:

# np.repeat behavior (copies are in succession, like aaabbbcccddd as described by topic starter)
einops.repeat(x, 'b c h w -> (b copy) c h w', copy=3)
# np.tile behavior (whole sequence is repeated 3 times like abcdabcdabcd)
einops.repeat(x, 'b c h w -> (copy b) c h w', copy=3)

You can repeat/tile multiple axes independently within one operation.

4 Likes
#10

man, really helpful. I’d like to say a great thanks to you

1 Like