Regarding broadcasting

1

according to broadcasting rules

If two tensors x , y are “broadcastable”, the resulting tensor size is calculated as follows:

  • If the number of dimensions of x and y are not equal, prepend 1 to the dimensions of the tensor with fewer dimensions to make them equal length.
  • Then, for each dimension size, the resulting dimension size is the max of the sizes of x and y along that dimension.
x = torch.randn(1, 1, 0)

y = torch.randn(4, 1, 1)

(x + y).shape

torch.Size([4, 1, 0])

according to the above rules, shouldn’t it have been 4, 1, 1, just y, or give an error?

2

This one give 4.1.2

x = torch.randn(1, 1, 1)
y = torch.randn(4, 1, 2)
(x+y).shape

and this one give an error

x = torch.randn(1, 1, 0)
y = torch.randn(4, 1, 2)
(x+y).shape
3

We can also take a look at the numpy broadcast rules, that state:

  1. they are equal, or
  2. one of them is 1

2 conditions satisfied in your example

>>> x_np = np.random.rand(4,1,0)
>>> y_np = np.random.rand(4,1,1)
>>> x_np + y_np
array([], shape=(4, 1, 0), dtype=float64)

dim of 0 is a special case, that will make your tensor volume to be 0.
Apparently, in is considered to be “bigger” than 1 in broadcasting. So the resulting dim is 0*1=0

4

but adding an empty tensor to some tensor, should return some tensor. I mean adding something to nothing should return something, why does it return nothing?
or it should give error, that addition of these two tensors is not possible.

5

It not actually like adding something to nothing, x = torch.randn(1, 1, 0) is empty indeed, but it has a shape.

Then, for each dimension size, the resulting dimension size is the max of the sizes of x and y along that dimension.

I don’t know where you read about this, but it’s incorrect. To broadcast, two must meet the rules, check Broadcasting semantics — PyTorch 2.1 documentation. For not equal dimensions, the resulting should be the one dimension which is not 1. (Generally the larger one, except for 0 I guess.

Take the broadcasting rules this way, if two dimension is not equal and one of them is 1, repeat the one with 1 dimension to make them match.

So when performing x + y with shape [1, 1, 0] and [4, 1, 1], image that

  • Along the first dimension, their shapes are not equal and x has dimension 1, so x is repeated 4 times, making it [4, 1, 0] (which is still empty).
  • Along the second dimension, their shapes are equal, so skip.
  • Along the third dimension, their shapes are not equal and y has dimension 1, so y is repeated 0 times, making it [4, 1, 0] (and it becomes empty).

Then we get a empty tensor with size [4, 1, 0].