Use Pytorch optimizer to minimize a user function

1

Dear all,
I have read many tutorials on how to use PyTorch to make a regression over a data set using for instance a model composed of several linear layers and the MSE loss.

Well, imagine that a known function F which depends on a variable x and some unknown parameters (p_j: j=0,…, P-1) with P relatively small, is a composition of special functions that cannot be modelized by the usual layers. So, my problem is a classical minimization problem knowing the data {x_i,y_i}_i<=N

 C(p_0,p_1,...,p_P) = Sum_i (F(x_i;{p_j}) - y_i)^2

The “loss” C({p_j}) that I want to minimize is the only function that I can call with a set of parameters.

Is there a way to use the PyTorch minimizers to get the minimum of C({p_j}) ?

2

Hi,

If F is implemented with pytorch Tensors and is differentiable. Yes you will be able to do it.

3

So, if I have just access to C({p_i}) through a python function, I cannot use the optimizers?

4

You don’t have access to {p_i} ?
You can do :

params = [p_0, p_1...]
opt = optim.SGD(params, lr=0.1)

for e in range(10):
  loss = C(params)
  opt.zero_grad()
  loss.backward()
  opt.step()
5

yes yes I can feed numerical values to C(ie the parameters to determine).
So, in your code snippet the params is a list of initial values eg.

params  = [0.1, 0.0001, -2., 1e3,...]

ok?

6

No, as I mentioned above, the function must work with pytorch Tensors.
So params = torch.tensor([0.1, 0.0001, -2., 1e3, ...], requires_grad=True) (or a list of Tensors as in my example.
Also, C must work with Tensors, if it converts it to python numbers or numpy arrays, gradients cannot be computed.

1 Like
7

ha yes, I understand, in F or C, the parameters must appear as torch.tensors. ok.

8

Hi @albanD and @Jean-Eric_Campagne , i am facing the same problem here, but the parameters are not being updated although i used requires grad = True.

    def minimize():
        xi = torch.tensor([1e-3, 1e-3, 1e-3, 1e-3, 1e-3, 1e-3], requires_grad=True)
        optimizer = torch.optim.Adam([xi], lr=0.1)
        for i in range(400):
            loss = self.f(xi)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        return xi

self.f(xi) is implemented in pytorch Tensors. Do you have any suggestions?

9

Hello,
I guess the experts will ask you to provide a complete working snippet of your example. try to keep it as simple as possible, ie. no class, just functions (ie. I see self.f so I imagine that you have extracted the code from a class). Then, expose your problem with some outputs for instance.

Then, I can run your example and try to reproduce your problem.

1 Like