Understanding Tensor Assignments

Say I create a placeholder for a batch of 3 images -

``````batch = torch.zeros(3, 3, 256, 256, dtype=torch.uint8)
``````

I have my dummy image -

``````image = torch.randint(size = (3,256,256), low=0, high=256)
``````

I then do -

``````batch[0] = image
``````

I am unable to understand the following outputs -

``````          id(batch[0]) == id(image)
out: False
``````

Should this not be true as both hold references to the same Tensor object ‘image’?

``````          id(batch.storage()) == id(image.storage())
out: True

batch[0][0][0][0] = 5
print(image[0][0][0])
out: 171

``````

Since both ‘batch’ and ‘image’ share the same underlying storage, why does the change in batch[0][0][0][0] not reflect when I print the corresponding element of ‘image’ ?

Thank you!

Hi Rohit!

Batch is a single “holistic” tensor of shape `[3, 3, 256, 256]`. It is
not a “collection” (in the sense of, say, a python list or dictionary) of
three 3x256x256 tensors.

Pytorch is doing some moderately fancy stuff with python here.

This is not simply assigning `image` to the `0` element of the `batch`
“collection.” I don’t understand the python details, but this line of
code, roughly speaking, calls something like:

``````batch.modify_tensor_slice (0, image)
``````

No, `batch[0]` and `image` are two different tensors.

In this case (unlike in the assignment `batch[0] = image`) you should
understand `batch[0]` as calling something like:

``````batch.return_slice_as_new_tensor (0)
``````

Pytorch tensors are fancy objects that can do a lot of things. In
this case, `batch[0]` is indeed a new tensor object, but it is a
“view,” so to speak, into another tensor, `batch`. But even though
the tensor `batch[0]` and the tensor `batch` share some of the
same underlying data, they are two distinct tensor objects.

This is a confusing fake-out. `batch` and `image` do not share the
same storage.

I don’t know the actual details of what is going on, but I deduce that
`some_tensor.storage()` returns a new “storage” object that wraps
(for whatever reason) the actual underlying storage.

Consider this:

``````>>> import torch
>>> torch.__version__
'1.7.1'
>>> batch = torch.tensor ([[1, 2], [3, 4]])
>>> image = torch.tensor ([10, 20])
>>> id (batch.storage())
139730615547904
>>> id (image.storage())
139730615547328
>>> id (batch.storage()) == id (image.storage())
True
>>> batch_storage = batch.storage()
>>> image_storage = image.storage()
>>> id (batch_storage) == id (image_storage)
False
>>> id (batch_storage)
139730615547904
>>> id (image_storage)
139730615546112
>>> id (batch.storage())
139730615547328
>>> id (image.storage())
139730615547584
``````

You can see that calling `.storage()` multiple times on the same
tensor returns a new, different “storage” object each time. (Why
pytorch does things this way I don’t know.)

But how then can we have:

``````>>> id (batch.storage()) == id (image.storage())
True
``````

Python / pytorch creates a new storage object when `image.storage()`
is called, gets its `id()` (address in memory), and then discards the
storage object. The same thing happens when `batch.storage()` is
called – a new storage object is created. Due to the vagaries of the
python interpreter the new storage object created by `batch.storage()`
happens to land at the same location in memory previously being
used by the since-discarded storage object created by
`image.storage()`, so the two have the same `id()`.

(Different python objects are only guaranteed to have different `id()`s
if they exist at the same time. Memory – and hence `id()`s – can be
reused after an object is disposed of.)

Finally, consider this:

``````>>> batch = torch.tensor ([[1, 2], [3, 4]])
>>> image = torch.tensor ([10, 20])
>>> b0 = batch[0]
>>> batch
tensor([[1, 2],
[3, 4]])
>>> image
tensor([10, 20])
>>> b0
tensor([1, 2])
>>> batch[0] = image
>>> batch
tensor([[10, 20],
[ 3,  4]])
>>> image
tensor([10, 20])
>>> b0
tensor([10, 20])
>>> batch[0, 0] = 55
>>> batch
tensor([[55, 20],
[ 3,  4]])
>>> image
tensor([10, 20])
>>> b0
tensor([55, 20])
>>> image[0] = 666
>>> batch
tensor([[55, 20],
[ 3,  4]])
>>> image
tensor([666,  20])
>>> b0
tensor([55, 20])
>>> b0[0] = 9999
>>> batch
tensor([[9999,   20],
[   3,    4]])
>>> image
tensor([666,  20])
>>> b0
tensor([9999,   20])
``````

To reiterate, `batch`, `image`, and `b0` are three distinct tensor objects.
But `batch` and `b0` share the same underlying storage (in that `b0`
is a “view” into `batch`), while `image` has its own, unrelated storage.

Best.

K. Frank

1 Like

Hello Frank!

Thank you very much for your detailed reply. Your answer has cleared all my doubts!

On another note, could you please elaborate a bit more on your first statement " Batch is a single ‘holistic’ tensor of shape [3, 3, 256, 256]. It is not a ‘collection’. " The mental picture I create of ‘Batch’ is a collection of 3 images each of shape [3,256,256]. I also understand that this is just really a single 4D tensor. Is this what you meant by a ‘holistic’ tensor ?

Thank you again!

Hi Rohit!

Well, first, you could consider this a semantic distinction about
what “collection” ought to mean. But leaving that aside …

Your mental picture is not unreasonable. It does have aspects,
however, that could be misleading.

From a low-level technical perspective, in the simplest case pytorch
stores the data for a tensor of shape `[3, 3, 256, 256]` as a
contiguous array of `3 * 3 * 256 * 256` (in your case) `byte`s.

Ignoring strides and various kinds of views, you can’t build a
`[3, 3, 256, 256]`-tensor from 3 `[3, 256, 256]`-tensors without
copying that data from the three separate tensors into the contiguous
data array for the new `[3, 3, 256, 256]`-tensor.

From the perspective of functionality, if the `batch` tensor were a
(general-purpose) collection of three `image` tensors, you might
imagine building a `batch` that consisted of a `[3, 256, 256]`-image,
a `[1, 256, 256]`-image, and a `[3, 128, 128]`-image. (Some
languages refer to such constructs as “ragged arrays.”) But you
can’t. The three images that make up the `batch` tensor all have to
have the same shapes. They are “slices” of a higher-dimensional
tensor (rather than items in a general-purpose collection), and, as
such, their shapes are constrained to be the same.

Best.

K. Frank

1 Like