Since both ‘batch’ and ‘image’ share the same underlying storage, why does the change in batch[0][0][0][0] not reflect when I print the corresponding element of ‘image’ ?
Batch is a single “holistic” tensor of shape [3, 3, 256, 256]. It is
not a “collection” (in the sense of, say, a python list or dictionary) of
three 3x256x256 tensors.
Pytorch is doing some moderately fancy stuff with python here.
This is not simply assigning image to the 0 element of the batch
“collection.” I don’t understand the python details, but this line of
code, roughly speaking, calls something like:
batch.modify_tensor_slice (0, image)
No, batch[0] and image are two different tensors.
In this case (unlike in the assignment batch[0] = image) you should
understand batch[0] as calling something like:
batch.return_slice_as_new_tensor (0)
Pytorch tensors are fancy objects that can do a lot of things. In
this case, batch[0] is indeed a new tensor object, but it is a
“view,” so to speak, into another tensor, batch. But even though
the tensor batch[0] and the tensor batch share some of the
same underlying data, they are two distinct tensor objects.
This is a confusing fake-out. batch and image do not share the
same storage.
I don’t know the actual details of what is going on, but I deduce that some_tensor.storage() returns a new “storage” object that wraps
(for whatever reason) the actual underlying storage.
Consider this:
>>> import torch
>>> torch.__version__
'1.7.1'
>>> batch = torch.tensor ([[1, 2], [3, 4]])
>>> image = torch.tensor ([10, 20])
>>> id (batch.storage())
139730615547904
>>> id (image.storage())
139730615547328
>>> id (batch.storage()) == id (image.storage())
True
>>> batch_storage = batch.storage()
>>> image_storage = image.storage()
>>> id (batch_storage) == id (image_storage)
False
>>> id (batch_storage)
139730615547904
>>> id (image_storage)
139730615546112
>>> id (batch.storage())
139730615547328
>>> id (image.storage())
139730615547584
You can see that calling .storage() multiple times on the same
tensor returns a new, different “storage” object each time. (Why
pytorch does things this way I don’t know.)
But how then can we have:
>>> id (batch.storage()) == id (image.storage())
True
Python / pytorch creates a new storage object when image.storage()
is called, gets its id() (address in memory), and then discards the
storage object. The same thing happens when batch.storage() is
called – a new storage object is created. Due to the vagaries of the
python interpreter the new storage object created by batch.storage() happens to land at the same location in memory previously being
used by the since-discarded storage object created by image.storage(), so the two have the same id().
(Different python objects are only guaranteed to have different id()s
if they exist at the same time. Memory – and hence id()s – can be
reused after an object is disposed of.)
To reiterate, batch, image, and b0 are three distinct tensor objects.
But batch and b0 share the same underlying storage (in that b0
is a “view” into batch), while image has its own, unrelated storage.
Thank you very much for your detailed reply. Your answer has cleared all my doubts!
On another note, could you please elaborate a bit more on your first statement " Batch is a single ‘holistic’ tensor of shape [3, 3, 256, 256]. It is not a ‘collection’. " The mental picture I create of ‘Batch’ is a collection of 3 images each of shape [3,256,256]. I also understand that this is just really a single 4D tensor. Is this what you meant by a ‘holistic’ tensor ?
Well, first, you could consider this a semantic distinction about
what “collection” ought to mean. But leaving that aside …
Your mental picture is not unreasonable. It does have aspects,
however, that could be misleading.
From a low-level technical perspective, in the simplest case pytorch
stores the data for a tensor of shape [3, 3, 256, 256] as a
contiguous array of 3 * 3 * 256 * 256 (in your case) bytes.
Ignoring strides and various kinds of views, you can’t build a [3, 3, 256, 256]-tensor from 3 [3, 256, 256]-tensors without
copying that data from the three separate tensors into the contiguous
data array for the new [3, 3, 256, 256]-tensor.
From the perspective of functionality, if the batch tensor were a
(general-purpose) collection of three image tensors, you might
imagine building a batch that consisted of a [3, 256, 256]-image,
a [1, 256, 256]-image, and a [3, 128, 128]-image. (Some
languages refer to such constructs as “ragged arrays.”) But you
can’t. The three images that make up the batch tensor all have to
have the same shapes. They are “slices” of a higher-dimensional
tensor (rather than items in a general-purpose collection), and, as
such, their shapes are constrained to be the same.