Minimizing a function in c++

1

Hi all.

I have a function f(X) which takes a 1d tensor and returns a float. I would like to attempt to find an X which minimizes f(X) using the pytorch c++ api.

I’ve seen a couple of mentions here of modelling the function with pytorch modules, but I’m having difficulty getting my head around the concept - can anyone help?

Thanks,
Ian

2

Here is my first attempt, however the SGD optimizer doesn’t like taking a tensor (where the tensor is the values I’m trying to optimise), failing with “no matching function for call to ‘torch::optim::SGD::SGD(at::Tensor&, double)”

torch::Tensor dynamic_parameters = torch::zeros({1}, torch::dtype(torch::kFloat64).requires_grad(true));

{
    torch::NoGradGuard guard;
    dynamic_parameters[0] = initial_guess;
}

torch::optim::SGD optimizer(dynamic_parameters, /*lr=*/0.01);

#define N_STEPS 20

for (int s = 0; s < N_STEPS; s ++)
{
    torch::Tensor result = torch::zeros({1}, torch::dtype(torch::kFloat64).requires_grad(true));

    torch::Tensor magnitudes = torch::zeros({N_V}, torch::dtype(torch::kFloat64).requires_grad(false));

    magnitudes += gState.magnitudes_ones_tensor * dynamic_parameters[0];

    torch::Tensor synth = gState.dots * magnitudes;

    torch::Tensor differences = torch::abs(real - synth);

    result[0] = torch::sum(differences);

    torch::Tensor loss = torch::nll_loss(result, prediction);

    loss.backward();

    optimizer.step();
}
3

Neither do the Python optimizers take a single tensor.
You can take pass in a vector of tensors, tough ({dynamic_parameters} might do the trick).

Best regards

Thomas

1 Like
4

Thank you, that compiles now.

I’m compiling a python module using pybind11, and now when I import my module (which imported ok before I started trying this pytorch stuff) I get:

importerror: undefined symbol: _ZTVN5torch5optim9OptimizerE

I’ll try to debug in c++, but I was wondering if you’d seen anything like that before?

5

I include pytorch in the .so with this:

#include <torch/torch.h>

But the .so that is created is 10 megabytes… I’m unclear whether it is actually including my built version of PyTorch? It seems unlikely.

6

No you need to link to it. The easiest is to use cmake.

1 Like
7

Thanks for your help Tom. This is my distutils script - is there an easy line to link it in to Python’s satisfaction?

from distutils.core import setup, Extension


def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)
      config.add_extension('reson8',
                           ['reson8.cpp'],
                           extra_info=info,
                           include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include/torch/csrc/api/include"])

      return config


if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)
8

Probably using the CppExtension/CUDAExtension helps set that up.

1 Like
9

I’ve changed my setup.py to use the CppExtension:

from distutils.core import setup, Extension
from torch.utils.cpp_extension import BuildExtension, CppExtension

def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)

      config.ext_modules.append(CppExtension(
                name='reson8',
                sources=['reson8.cpp'],
                extra_info=info,
                extra_compile_args=['-g'],
                include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include/torch/csrc/api/include"]
                                ))

      return config


if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)

But I get a similar error:

ImportError: reson8.so: undefined symbol: _ZTIN3c1021AutogradMetaInterfaceE

So there’s still a build problem…

10

Well, it builds fine, it just isn’t including something it needs.

I also added the pytorch include directory to LD_LIBRARY_PATH, but that didn’t help.

11

If you get the error on import, you might have a version mismatch somewhere from having several PyTorch versions lying around?

1 Like
12

I suppose… but I don’t know how to fix that easily. The .so should be using the version of pytorch from the c++ import, not the current PIP version right.

13

My standard approach is to use the libtorch copy that comes with the Python packages for compilation, but I only have self-built PyTorch…

1 Like
14

I have self-built pytorch - can you tell me exactly what you do?

15

I managed to fix this in the end - I posted the answer here:

https://stackoverflow.com/questions/65774829/how-do-i-link-in-all-of-pytorch-with-a-pybind11-so/65828949#65828949