What ops make a tensor non contiguous?

1

Hey

I’ve noticed expand/expand_as makes a tensor non-contiguous

Is there any other op that might do this?

Thanks

Henry

2 Likes
2

AFAIK there is

  • t() transpose
  • some Select/Slice operations, especially those with stride>1, i.e. tensor[::2]
  • expand

In such cases, the storage doesn’t change, it only modifies stride(you can get it by tensor.stride()).

is_contiguous is implemented in C.

int THTensor_(isContiguous)(const THTensor *self)
{
  long z = 1;
  int d;
  for(d = self->nDimension-1; d >= 0; d--)
  {
    if(self->size[d] != 1)
    {
      if(self->stride[d] == z)
        z *= self->size[d];
      else
        return 0;
    }
  }
  return 1;
}
6 Likes
3

Thanks,

So how is the contiguous() function implemented then?

Can you point me to the location of the source?

Thanks,

Henry

4

It just copy the data and make a new tensor. See implementation in TH

https://github.com/pytorch/pytorch/blob/master/torch/lib/TH/generic/THTensor.c#L182-L199

5

Gotcha, so when using expand should I always do a contiguous() after it or does that only add extra computation?

6

In fact, I never add contiguous after expand. I often use expand for something like broadcast in numpy, after the broadcast operation, the result should be collect and is contiguous.

import torch as t
a = t.Tensor(3,4)
b = t.Tensor(3)
c = b.unsqueeze(1).expand_as(a) + a
print(c.is_contiguous())

expand is a memory-efficient operation because it won’t copy data in memory(repeat will do ), if you make it contiguous, it will copy data and occupy extra memory.

import torch as t
a = t.Tensor(1,3)
b = a.expand(100000000000,3)
print(id(a.storage())==id(b.storage()))
print(b.storage())
# !!!!dont't do below!!!!!
# Print(b) or b.contiguous()
1 Like
7

Gotcha.

I’ll stick to not using contiguous then. Thanks :slight_smile:

8

According to this, following operators makes tensor non-contiguous:

narrow() , view() , expand() or transpose()