Pytorch tensor stride - how it works

1

PyTorch doesn’t seem to have documentation for tensor.stride().
Can someone confirm my understanding?

My questions are three-fold.

  1. Stride is for accessing an element in the storage. So stride size will be the same as the dimension of the tensor. Correct?

  2. For each dimension, the corresponding element of stride tells how much it takes to move along the 1-dimensional storage.

For example:

In [15]: x = torch.arange(1,25)

In [16]: x
Out[16]:
tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18
, 19, 20, 21, 22, 23, 24])

In [17]: a = x.view(4,3,2)

In [18]: a
Out[18]:
tensor([[[ 1,  2],
         [ 3,  4],
         [ 5,  6]],

        [[ 7,  8],
         [ 9, 10],
         [11, 12]],

        [[13, 14],
         [15, 16],
         [17, 18]],

        [[19, 20],
         [21, 22],
         [23, 24]]])

In [20]: a.stride()
Out[20]: (6, 2, 1)
  1. How does having this information help perform tensor operations efficiently? Basically this is showing the memory layout. So how does it help?
2 Likes
2

  1. The stride will have the same number of values as the number of dimensions. E.g. if you are dealing with a tensor with 4 dimensions, tensor.stride() will return 4 values.

  2. Yes, that’s correct and this post gives another example with contiguous vs. non-contiguous tensors.

  3. The stride is used in the backend for indexing, which can be used if you want to directly access specific elements in the memory block.

5 Likes