Hey

I’ve noticed expand/expand_as makes a tensor non-contiguous

Is there any other op that might do this?

Thanks

Henry

Hey

I’ve noticed expand/expand_as makes a tensor non-contiguous

Is there any other op that might do this?

Thanks

Henry

2 Likes

AFAIK there is

- t() transpose
- some Select/Slice operations, especially those with stride>1, i.e. tensor[::2]
- expand

In such cases, the storage doesn’t change, it only modifies `stride`

(you can get it by `tensor.stride()`

).

`is_contiguous`

is implemented in C.

```
int THTensor_(isContiguous)(const THTensor *self)
{
long z = 1;
int d;
for(d = self->nDimension-1; d >= 0; d--)
{
if(self->size[d] != 1)
{
if(self->stride[d] == z)
z *= self->size[d];
else
return 0;
}
}
return 1;
}
```

6 Likes

Thanks,

So how is the contiguous() function implemented then?

Can you point me to the location of the source?

Thanks,

Henry

It just copy the data and make a new tensor. See implementation in TH

https://github.com/pytorch/pytorch/blob/master/torch/lib/TH/generic/THTensor.c#L182-L199

Gotcha, so when using expand should I always do a contiguous() after it or does that only add extra computation?

In fact, I never add `contiguous`

after expand. I often use `expand`

for something like `broadcast`

in numpy, after the `broadcast`

operation, the result should be collect and is contiguous.

```
import torch as t
a = t.Tensor(3,4)
b = t.Tensor(3)
c = b.unsqueeze(1).expand_as(a) + a
print(c.is_contiguous())
```

`expand`

is a memory-efficient operation because it won’t copy data in memory(`repeat`

will do ), if you make it contiguous, it will copy data and occupy extra memory.

```
import torch as t
a = t.Tensor(1,3)
b = a.expand(100000000000,3)
print(id(a.storage())==id(b.storage()))
print(b.storage())
# !!!!dont't do below!!!!!
# Print(b) or b.contiguous()
```

1 Like

Gotcha.

I’ll stick to not using contiguous then. Thanks

According to this, following operators makes tensor non-contiguous:

`narrow() , view() , expand() or transpose()`