What’s the proper way to do constrained optimization in PyTorch?
For example, I want each parameter of my model to be bounded both from above and below by some constants cLow and cHigh.
That is, if W is the d-dimensional (flattened) weight vector of my model, I’d like to enforce
cLow < W[i] < cHigh for i = 1, 2, … d. How can I do that?
You can do projected gradient descent by enforcing your constraint after each optimizer step. An example training loop would be:
opt = optim.SGD(model.parameters(), lr=0.1)
for i in range(1000):
out = model(inputs)
loss = loss_fn(out, labels)
print(i, loss.item())
opt.zero_grad()
loss.backward()
opt.step()
with torch.no_grad():
for param in model.parameters():
param.clamp_(-1, 1)
The last three lines enforce the constraint that the weights fall in the range -1–1.
I am also working on constraints optimization problems. My experience with this suggestion is not positive. When I don’t use clamp_() and train model with no restriction then value of specific weights of interested is close to desired and model is predicting good results. But after using clamp_(), model performance degrade severely. What can be the possible reason for this.
One reason is that this clamping is not communicated to the optimizer, and in particular destroys the gradients. So the optimizer falsely believes that it has moved the parameter in a certain direction, when in fact it is clamped to the same value as before.
@oricha Did you find any solution to that problem?
I’ve worked quite a bit on combining PyTorch with constrained optimization. On one hand, my constraints are nonlinear functions of the parameters, as opposed to bound constraints here. The temporary solution I decided upon was using SciPy’s optimize routines, and using PyTorch to compute gradients.
The code for this was part of a more general paper, but the main function is here.
Another option would be a similar (more professional) wrapper of PETSc: