LBFGS Optimizer is non-deterministic?

samuelsunarjo96 · April 13, 2020, 4:20pm

LBFGS Optimizer:

pytorch/pytorch/blob/6be85603757a810be3d18b70279b104c0653e9b3/torch/optim/lbfgs.py#L183


        if abs(bracket[1] - bracket[0]) * d_norm < tolerance_change:
            break


    # return stuff
    t = bracket[low_pos]
    f_new = bracket_f[low_pos]
    g_new = bracket_g[low_pos]
    return f_new, g_new, t, ls_func_evals




class LBFGS(Optimizer):
    """Implements L-BFGS algorithm, heavily inspired by `minFunc
    <https://www.cs.ubc.ca/~schmidtm/Software/minFunc.html>`.


    .. warning::
        This optimizer doesn't support per-parameter options and parameter
        groups (there can be only one).


    .. warning::
        Right now all parameters have to be on a single device. This will be
        improved in the future.

I was wondering if this optimizer is supposed to produce non-deterministic results? Given that these results were produced under the same parameter initialization and optimization setting. Simply running the program multiple times without any changes would produce different results.

I have set the following to attempt disabling the non-deterministic outcome but no luck:
np.random.seed(0)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

albanD · April 13, 2020, 6:15pm

Hi,

You can check the note on reproducibility in the doc here: https://pytorch.org/docs/stable/notes/randomness.html

There are many reasons that could lead to it (or any other part of your code) to be non deterministic.