How to implement a customize weight

Hi, I would like to implement tanh(weight) instead of weight, for example:
I am not sure if there is a problem with my code,so thank you so much for your help.

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 6, 3, 1, 2,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(6, 16, 5,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.fc1 = nn.Linear(16 * 5 * 5, 120,bias=False)
        self.fc2 = nn.Linear(120, 10,bias=False)
        self.conv1.weight =torch.tanh(self.conv1.weight)
        self.conv2.weight =torch.tanh(self.conv2.weight)
        self.fc1.weight =torch.tanh(self.fc1.weight)
        self.fc2.weight =torch.tanh(self.fc2.weight)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size()[0], -1)
        x = self.fc1(x)
        x= F.relu(x)
        x = self.fc2(x)
        return x

Do you mean weight is learnable and calculate y = tanh(weight) * x when learning?

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 6, 3, 1, 2,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(6, 16, 5,bias=False),
            nn.ReLU(),
            nn.MaxPool2d(2, 2)
        )
        self.fc1 = nn.Linear(16 * 5 * 5, 120,bias=False)
        self.fc2 = nn.Linear(120, 10,bias=False)

    def forward(self, x):
        with torch.no_grad():
            self.conv1[0].weight[:] = torch.tanh(self.conv1[0].weight[:])
            self.conv2[0].weight[:] = torch.tanh(self.conv2[0].weight[:])
            self.fc1.weight[:] = torch.tanh(self.fc1.weight[:])
            self.fc2.weight[:] = torch.tanh(self.fc2.weight[:])
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size()[0], -1)
        x = self.fc1(x)
        x= F.relu(x)
        x = self.fc2(x)
        return x

I am not sure with torch.no_grad is needed or not.

1 Like

Thank you for your kind reply. Yes, weight is a learnable parameter, and use tanh(weight) to limit the weight range.
Apart from with torch.no_grad, what is the difference between putting self.conv1[0].weight[:] = torch.tanh(self.conv1[0].weight[:]) before def forward(self, x): and inside def forward(self, x):.

Thank you again for your time.
Best regards.

__init__ only work when you init your network.
If you want to limit it, inside forward
.

oh! Thank you so much!

If all you want is to limit the values to be in a range, why not using the torch.clamp?
I just want to make sure you use the right tools here, tanh is an activation function, and it transforms your input to be in that range, depending on the task it might not give you what you desire.
So if you know what you are doing then fine, in case you didn’t know, check torch.clamp out, maybe this is what you were looking for?

Yes, if ONLY CLIP, check here

1 Like

Hi, thank you for your suggestion. I have tried torch.clamp, besides, I also want to use tanh (weight) for weight regularizer for loss function, for example:
image

Thank you so much for all.

1 Like