Is optim.LBFGS GPU-based?

1

The Scipy library has an optimization method called Basinhopping which we can optimize an arbitrary function and extract variables which minimize its function’s value. We also can combine this method with L-BFGS. However, the problem is that, it is indeed extremely slow, and the main issue is that Scipy is single-threaded.
I know that PyTorch does have a L-BFGS algorithm. Does this PyTorch version of L-BFGS support multi-threading and GPU-based? (In fact, I want to be able to get an enormous speedup by using that.)
Moreover, can I optimize a function and find its corresponding variables that minimize the function with it?

2

optim.LBFGS will optimize the Variables you pass in on a GPU; there is nothing in the source code that would suggest otherwise.

Let’s say you’re optimizing over the Variables x and a function f(x). Try something like the following:

import torch
from torch.autograd import Variable
from torch.optim import LBFGS

x = Variable(torch.Tensor([100]).cuda(), requires_grad=True)
optimizer = LBFGS([x])
def f(x):
	return (x - 1) * (x - 1)

n_step = 2

for _ in range(0, n_step):
    def closure():
        optimizer.zero_grad()
        output = f(x)
        output.backward()
        return output
    optimizer.step(closure)
    print(x)
2 Likes