How to apply constraint to individual weights?

1

Hello, I’m trying to implement Lotka Volterra system from this paper. Looking at eq 3. they are enforcing some constraints for each parameter. I was curios I how I achieve a similar behavior? For example some weights to be always in [0,1] interval, or another weight matrix, to always have 1. on the diagonal.

2

Lets say you have a constraint between 0 and 1, you can do it the following way

# Constraints
from torch import nn
from torch.autograd import Variable
import torch.nn.functional as F
import torch

torch.manual_seed(42)

class model1(nn.Module):
  def __init__(self):
    super(model1,self).__init__()
    self.l1=nn.Linear(100,10)
    self.l2=nn.Linear(10,5)
    self.l3=nn.Linear(5,1)

  def forward(self,x):
    out=self.l1(x)
    out=self.l2(out)
    out=self.l3(out)
    return(out)

model=model1()
criterion=nn.MSELoss()
optimizer=optim.Adam(model.parameters(),lr=0.001)

X=Variable(torch.from_numpy(np.random.rand(1000,100).astype(np.float32)))
Y=Variable(torch.from_numpy(np.random.randint(2,size=(1000)).astype(np.float32)))
epochs=1000

constraintHigh=1
constraintLow=0

for curEpoch in range(epochs):
  model.zero_grad()
  output=model(X)
  loss=criterion(output,Y)
  loss.backward()
  optimizer.step()
  # We will now apply the constraint and reset the weights
  model.l1.weight=torch.nn.Parameter(constraintLow + (constraintHigh-constraintLow)*(model.l1.weight - torch.min(model.l1.weight))/(torch.max(model.l1.weight) - torch.min(model.l1.weight)))
  if(curEpoch % 100==0):
    print("The loss is {0}".format(loss.item()))
    print("Weight Min is {0} and Max is {1}".format(torch.min(model.l1.weight),torch.max(model.l1.weight)))

Hope this helps

1 Like
3

Thank you for the solution.

I have tried this way, but I got bad regression results. I assume that the weight and bias are modified, and away from the origin value. I think the origin value is correct value by BP.

Do you have any solution of this issue?
Thank you very much.