How do pytorch know, if it is single input/multiple input

1

I have a simple question

Lets assume we have a Linear Layer like

layer = nn.Linear(12 , 12)

If we pass something like

inp = torch.rand(12)

layer(inp)

We get a tensor of size torch.Size([12])

And if we pass input like

inp = torch.rand(13 , 12)

layer(inp)

We get a tensor of size torch.Size([13 , 12])
So how do Pytorch actually knows if it is

  • Single Input
    or
  • Multi Input

Is it a simple if-else condition followed by iterating for loop or broadcasting, or anything totaly different ā€¦?

1 Like
2

That is how matrix multiplication works:

1  4 
          w1
2  5                =    [w1*1 + w2*4, 2*w1 + 5*w2,...]
          w2
3  6

Here the rectangle of numbers is your input tensor, where each row is a sample. It is only one row for a single sample (1,12) in your example.

The linear layer is a column vector w = < w1, w2 > where w1 and w2 are the weights.

Each row in the numeric matrix is a sample (1, 4), (2, 5)ā€¦

So all you need to do is iterating over whichever tensor a user gives as input, and this includes (1,4)

And carry out the multiplication between each row and the tensor, in this case you get:
Results = [w1*1 + w2*4, 2*w1 + 5*w2,...]

As to the exact code implementation I am not sure, but conceptually that is how you do it, and does not matter the number of samples in the input.

1 Like
3

So is it if-else condition first followed by for loop like

if multi_input : # 1 Extra Dimension then expected number of dimensions, here (13 , 12) ---> 13 is extra
    # iterate over the list and caluclate the matmul for every value and append in the output list
else : # simply calculate the matmul
4

From my understanding, in the case of 1D matmul computes the dot product because those are now vectors.

See the docs here., first bullet point.

Also, the remaining explanation should help, I include it for easy of access:

  • If both tensors are 1-dimensional, the dot product (scalar) is returned.
  • If both arguments are 2-dimensional, the matrix-matrix product is returned.
  • If the first argument is 1-dimensional and the second argument is 2-dimensional, a 1 is prepended to its dimension for the purpose of the matrix multiply. After the matrix multiply, the prepended dimension is removed.
1 Like
5

Okay got it, was a little confused. Thanks mate :slight_smile:

1 Like
6

you are welcome, feel free to message me to chat, iā€™m learning as well

1 Like