Is the weighted sum calculated by transposing the weights associated with a linear layer under the hood in a torch nn model?

1

In the simple nn module as shown below, the shape of the weights associated with fc1, i.e. W1, is (128 x 784).

Assuming the mini batch size is 64, so the shape of the input X is (64, 784).

Does this mean that under the hood the weighted sum calculation inside fc1 is carried out as the dot product between input X (shape: 64 x 784) and the transpose of W1 (784 x 128) to obtain an output for fc1 with a shape of (64 x 128)?

from torch import nn
import torch.nn.functional as F 

class myNN(nn.Module):
    def __init__(self):
        super().__init__()
        # the first hidden layer
        self.fc1 = nn.Linear(784, 128)
        # the second hidden layer
        self.fc2 = nn.Linear(128, 64)
        # the output layer 
        self.fc3 = nn.Linear(64, 10)
        
    def forward(self, x):
        # the 1st hidden layer
        x = F.relu(self.fc1(x))
        # the 2nd hidden layer 
        x = F.relu(self.fc2(x))
        # the output layer 
        x = F.softmax(self.fc3(x), dim=1)
        
        return x

# create the nn object
model = myNN()

print('fc1 weights shape:\n', model.fc1.weight.shape)

The output is:

fc1 weights shape:
 torch.Size([128, 784])
2

The forward method in nn.Linear is defined as y = xW^T + b, so yes the weight matrix is transposed in the operation. You can read more in the docs,

https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear

1 Like