Regarding broadcasting

according to broadcasting rules

If two tensors x , y are “broadcastable”, the resulting tensor size is calculated as follows:

  • If the number of dimensions of x and y are not equal, prepend 1 to the dimensions of the tensor with fewer dimensions to make them equal length.
  • Then, for each dimension size, the resulting dimension size is the max of the sizes of x and y along that dimension.
x = torch.randn(1, 1, 0)

y = torch.randn(4, 1, 1)
(x + y).shape

torch.Size([4, 1, 0])

according to the above rules, shouldn’t it have been 4, 1, 1, just y, or give an error?

This one give 4.1.2

x = torch.randn(1, 1, 1)
y = torch.randn(4, 1, 2)
(x+y).shape

and this one give an error

x = torch.randn(1, 1, 0)
y = torch.randn(4, 1, 2)
(x+y).shape

We can also take a look at the numpy broadcast rules, that state:

  1. they are equal, or
  2. one of them is 1

2 conditions satisfied in your example

>>> x_np = np.random.rand(4,1,0)
>>> y_np = np.random.rand(4,1,1)
>>> x_np + y_np
array([], shape=(4, 1, 0), dtype=float64)

dim of 0 is a special case, that will make your tensor volume to be 0.
Apparently, in is considered to be “bigger” than 1 in broadcasting. So the resulting dim is 0*1=0

but adding an empty tensor to some tensor, should return some tensor. I mean adding something to nothing should return something, why does it return nothing?
or it should give error, that addition of these two tensors is not possible.

It not actually like adding something to nothing, x = torch.randn(1, 1, 0) is empty indeed, but it has a shape.

Then, for each dimension size, the resulting dimension size is the max of the sizes of x and y along that dimension.

I don’t know where you read about this, but it’s incorrect. To broadcast, two must meet the rules, check https://pytorch.org/docs/stable/notes/broadcasting.html. For not equal dimensions, the resulting should be the one dimension which is not 1. (Generally the larger one, except for 0 I guess.

Take the broadcasting rules this way, if two dimension is not equal and one of them is 1, repeat the one with 1 dimension to make them match.

So when performing x + y with shape [1, 1, 0] and [4, 1, 1], image that

  • Along the first dimension, their shapes are not equal and x has dimension 1, so x is repeated 4 times, making it [4, 1, 0] (which is still empty).
  • Along the second dimension, their shapes are equal, so skip.
  • Along the third dimension, their shapes are not equal and y has dimension 1, so y is repeated 0 times, making it [4, 1, 0] (and it becomes empty).

Then we get a empty tensor with size [4, 1, 0].