How do pytorch know, if it is single input/multiple input

I have a simple question

Lets assume we have a Linear Layer like

``````layer = nn.Linear(12 , 12)
``````

If we pass something like

``````inp = torch.rand(12)

layer(inp)
``````

We get a tensor of size `torch.Size([12])`

And if we pass input like

``````inp = torch.rand(13 , 12)

layer(inp)
``````

We get a tensor of size `torch.Size([13 , 12])`
So how do Pytorch actually knows if it is

• Single Input
or
• Multi Input

Is it a simple `if-else condition` followed by `iterating for loop` or `broadcasting`, or anything `totaly different` ā¦?

1 Like

That is how matrix multiplication works:

``````1  4
w1
2  5                =    [w1*1 + w2*4, 2*w1 + 5*w2,...]
w2
3  6
``````

Here the rectangle of numbers is your input tensor, where each row is a sample. It is only one row for a single sample (1,12) in your example.

The linear layer is a column vector `w = < w1, w2 >` where w1 and w2 are the weights.

Each row in the numeric matrix is a sample (1, 4), (2, 5)ā¦

So all you need to do is iterating over whichever tensor a user gives as input, and this includes (1,4)

And carry out the multiplication between each row and the tensor, in this case you get:
`Results = [w1*1 + w2*4, 2*w1 + 5*w2,...]`

As to the exact code implementation I am not sure, but conceptually that is how you do it, and does not matter the number of samples in the input.

1 Like

So is it `if-else condition` first followed by `for loop` like

``````if multi_input : # 1 Extra Dimension then expected number of dimensions, here (13 , 12) ---> 13 is extra
# iterate over the list and caluclate the matmul for every value and append in the output list
else : # simply calculate the matmul
``````

From my understanding, in the case of 1D `matmul` computes the dot product because those are now vectors.

See the docs here., first bullet point.

Also, the remaining explanation should help, I include it for easy of access:

• If both tensors are 1-dimensional, the dot product (scalar) is returned.
• If both arguments are 2-dimensional, the matrix-matrix product is returned.
• If the first argument is 1-dimensional and the second argument is 2-dimensional, a 1 is prepended to its dimension for the purpose of the matrix multiply. After the matrix multiply, the prepended dimension is removed.
1 Like

Okay got it, was a little confused. Thanks mate

1 Like

you are welcome, feel free to message me to chat, iām learning as well

1 Like