I have the following code for a neural network. I am confused about what is the difference between the use of init() and forward() methods. Does the init() method behave as the constructor? If so, what is the significance of the forward() method? Is it necessary to use both while creating the network?
class MyNeuralNetwork(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(MyNeuralNetwork, self).__init__()
self.fc1 = nn.Linear(input_size, hidden_size)
self.relu = nn.ReLU()
self.fc2 = nn.Linear(hidden_size, output_size)
self.sigmoid = nn.Sigmoid()
def forward(self, x):
out = self.fc1(x)
out = self.relu(out)
out = self.fc2(out)
out = self.sigmoid(out)
return out
__init__ is a constructor method used to initialize the parameters of the network. It is executed when an object of the class is created. For example, in PyTorch, this method is used to define the layers of the network, such as convolutional layers, linear layers, activation functions, etc.
forward is the method that defines the forward pass of the neural network. This method takes the input data and passes it through the layers of the network to produce the output. This method is executed whenever the model is called to make a prediction or to compute the loss during training.
In other words, __init__ sets up the network’s structure by defining the layers, while forward specifies how the data flows through the network. Both methods are required to create a neural network in PyTorch and serve different purposes.
If I need to create a tensor based on the arguments in the constructor, should I create this tensor in the constructor or in def forward()? This tensor won’t change with the input data for forward(). Is it better to create it in the constructor because we don’t need to create it again and again for each batch? Thanks!
Yes, if you need to create a tensor, based on arguments that do not change during the forward pass and remain constant throughout the lifetime of the module, it is generally better to create this tensor in the constructor (__init__ ) rather than in the forward method. This is because the constructor is called only once when you initialize the model, whereas the forward method is called for each forward pass, which can be for every batch of data during training or inference.