Beginner: Should ReLU/sigmoid be called in the __init__ method?

nlp
13

Oh, Thanks!
so the learnable parameters should be held in the _init_ part within MyModel. So during the forward and backward process, these parameters could be updated progressively. So in this way, MyModel can be either written in this way:

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv_like = nn.convlike() # which has its own hidden parameters 
        
    def forward(self, x):
        x = self.conv_like(x)

or written in this way:

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.func_params = params # which will be used in nn.functional.funs 
        
    def forward(self, x):
        x = nn.functional.funs(x,self.func_params)

other nn.funs or nn.functional.funs which do not have hidden learnable parameters could be placed wherever you like.
is it correct?

6 Likes