Rearrange 1d tensor with overlapping

1

I read the unfold documentation but I fail to understand how to use it for my following case.
I have the following case where
tmp is a 1D tensor and step is the overlapping parameter.

I have the following loop that concat the tmp based on the step (ovedrlap) as follow

N = 10
tmp = torch.arange(0,16*N).reshape(1, 16,N ) 
step=2 # this can change to 1,3,.ect
for i in range(N):
     print(i*step,(i+1)*step+(step//2)*2)
tmp = torch.cat([tmp[:,:, i*step:(i+1)*step+(step//2)*2] for i in range(N) if (i+1)*step+(step//2)*2 < tmp.shape[2]], dim=0)

is there a way to do such a task via unfolding? or any other way?

the original tmp looks like this:

tensor([[[  0,   1,   2,   3,   4,   5,   6,   7,   8,   9],
         [ 10,  11,  12,  13,  14,  15,  16,  17,  18,  19],
         [ 20,  21,  22,  23,  24,  25,  26,  27,  28,  29],
         [ 30,  31,  32,  33,  34,  35,  36,  37,  38,  39],
         [ 40,  41,  42,  43,  44,  45,  46,  47,  48,  49],
         [ 50,  51,  52,  53,  54,  55,  56,  57,  58,  59],
         [ 60,  61,  62,  63,  64,  65,  66,  67,  68,  69],
         [ 70,  71,  72,  73,  74,  75,  76,  77,  78,  79],
         [ 80,  81,  82,  83,  84,  85,  86,  87,  88,  89],
         [ 90,  91,  92,  93,  94,  95,  96,  97,  98,  99],
         [100, 101, 102, 103, 104, 105, 106, 107, 108, 109],
         [110, 111, 112, 113, 114, 115, 116, 117, 118, 119],
         [120, 121, 122, 123, 124, 125, 126, 127, 128, 129],
         [130, 131, 132, 133, 134, 135, 136, 137, 138, 139],
         [140, 141, 142, 143, 144, 145, 146, 147, 148, 149],
         [150, 151, 152, 153, 154, 155, 156, 157, 158, 159]]])

and the final version looks like this (shape=torch.Size([3, 16, 4])):

tensor([[[  0,   1,   2,   3],
         [ 10,  11,  12,  13],
         [ 20,  21,  22,  23],
         [ 30,  31,  32,  33],
         [ 40,  41,  42,  43],
         [ 50,  51,  52,  53],
         [ 60,  61,  62,  63],
         [ 70,  71,  72,  73],
         [ 80,  81,  82,  83],
         [ 90,  91,  92,  93],
         [100, 101, 102, 103],
         [110, 111, 112, 113],
         [120, 121, 122, 123],
         [130, 131, 132, 133],
         [140, 141, 142, 143],
         [150, 151, 152, 153]],

        [[  2,   3,   4,   5],
         [ 12,  13,  14,  15],
         [ 22,  23,  24,  25],
         [ 32,  33,  34,  35],
         [ 42,  43,  44,  45],
         [ 52,  53,  54,  55],
         [ 62,  63,  64,  65],
         [ 72,  73,  74,  75],
         [ 82,  83,  84,  85],
         [ 92,  93,  94,  95],
         [102, 103, 104, 105],
         [112, 113, 114, 115],
         [122, 123, 124, 125],
         [132, 133, 134, 135],
         [142, 143, 144, 145],
         [152, 153, 154, 155]],

        [[  4,   5,   6,   7],
         [ 14,  15,  16,  17],
         [ 24,  25,  26,  27],
         [ 34,  35,  36,  37],
         [ 44,  45,  46,  47],
         [ 54,  55,  56,  57],
         [ 64,  65,  66,  67],
         [ 74,  75,  76,  77],
         [ 84,  85,  86,  87],
         [ 94,  95,  96,  97],
         [104, 105, 106, 107],
         [114, 115, 116, 117],
         [124, 125, 126, 127],
         [134, 135, 136, 137],
         [144, 145, 146, 147],
         [154, 155, 156, 157]]])
2 
a = torch.arange(160).view(1, 16, 10)
tmp = a[0].unfold(1, 6, 1)[:,:4,::2].permute(2, 0, 1)
print(tmp)

Not that I’d know why. :slight_smile:

Best regards

Thomas

1 Like
3

Hi Thomas

Thanks a lot for your answer! It is actually doing it correctly and much faster. I dont know how did you figure it out, but I really appreciate it anyways.
can I ask what is the strategy behind choosing 6 and :4 and ::2 in .unfold(1, 6, 1)[:,:4,::2] like how should they change for another step (e.g 8).

Just since you asked why I want to do it, I am trying to develop something similar to slide window for attention. but I am not sure if it gonna work yet :smiley:

4

So I knew Tensor.unfold(dim, size, step) (which is different from nn.functional.unfold) can produce the type of overlapping patches you want. Initially, I thought it would be step 2 but this will make the 0 2 4 6, so we need step 1 and then skip every other slice with the ::2. The size = 6 is because you asked for 3 outputs but as we are skipping every other we need 6 (probably 5 would do in hindsight). Then the :4 is to limit each row to 4 elements.

There also is the - perhaps more straightforward alternative - to use a.as_strided((3, 16, 4), (2, 10, 1)) to specify the layout in memory directly but I am never sure how much to trust as_strided with autograd. Quite likely, everything just works, but at some point in the distant past, it did something that I found strange. In that sense Tensor.unfold seemed more natural even if it looks like black magic with all these numbers derived through iterative fiddeling.

Best regards

Thomas

2 Likes
5

Thanks a lot for the clarification! I think I understand it now much better :slight_smile:

I was wondering if you any suggestion for this case if now I want to use fold to make it back.
Hmmm I think we also dont have x.fold in pytorch?