I’m trying to find a way to make a slice of a tensor contiguous. One way I tried is the following using permuate.
import torch
x = torch.zeros([16, 8, 1024, 128], device='cuda:0')
print(x[:,:, 128:256].is_contiguous()) #False
y = torch.zeros([1024, 16, 8, 128], device='cuda:0')
print(y[128:256].is_contiguous()) #True
z = y.permute(1,2,0,3)
assert z.size()==x.size()
print(z[:,:,128:256].is_contiguous()) #False
Why is the last print False? I confirmed the data_ptr() still the same, so assume no copy has been made. If this is not the right way to get a contiguous slice, can someone share a trick for that?
Call .contiguous() on the slice in question. If the slice is already contiguous,
it will be a no-op. If not, it will return a contiguous copy of the slice.
Taking a (non-trivial) permutation of a (contiguous) tensor makes it
non-contiguous.
There are two things going on here. Slicing a set of rows (that is, its first
dimension) out of a tensor gives you a contiguous subtensor, while slicing
out a set of columns (or anything other than the first dimension) gives you
a non-contiguous tensor.
This is the key issue.
But you’re trying to fix this issue by using .permute() to make your slice
apply to the first dimension. However, .permute(), itself, renders the tensor
non-contiguous.
The subtensor (slice) you want simply isn’t stored contiguously in memory
so you have to make a contiguous copy of the data to get your contiguous
slice. .contiguous() does this for you. (I wouldn’t call this a trick – it just
makes a brute-force copy.)
Consider:
>>> import torch
>>> torch.__version__
'2.1.2'
>>> t = torch.zeros (4, 4)
>>> t.is_contiguous()
True
>>> t[1:3, :].is_contiguous() # the two rows in the slice are still contiguous
True
>>> t[:, 1:3].is_contiguous() # not contiguous -- have to jump over the missing columns
False
>>> t.permute (1, 0).is_contiguous() # no longer row-major -- not contiguous
False
>>> t[:, 1:3].contiguous().is_contiguous() # .contiguous() creates a new contiguous copy
True
Thanks KFrank. Actually, I’m trying to avoid calling “contiguous()” for the slice, as it becomes expensive in my case (which means every slice I create is NOT contiguous()).