optim.LBFGS calls closure 20 times for each step

vision
#1

I’m going through the tutorial at style transfer.

One thing I can’t figure out is it seems every time the optimizer.step(closure) statement gets executed, the closure function is called 20 times. Can someone please explain why?

Steps to reproduce:
0. Run the ipython notebook ipython notebookuntil the last cell.

  1. Add one line to print the run[0] variable in the closure() function:
def closure():
    # correct the values of updated input image
    input_param.data.clamp_(0, 1)

    optimizer.zero_grad()
    model(input_param)
    style_score = 0
    content_score = 0

    for sl in style_losses:
        style_score += sl.backward()
    for cl in content_losses:
        content_score += cl.backward()

    run[0] += 1
    print("run {}:".format(run))
    if run[0] % 50 == 0:
        print("run {}:".format(run))
        print('Style Loss : {:4f} Content Loss: {:4f}'.format(
            style_score.data[0], content_score.data[0]))
        print()

    return style_score + content_score
  1. Run the following:
run = [0]
input_param, optimizer = get_input_param_optimizer(input_img)
optimizer.step(closure)
run [1]:
run [2]:
run [3]:
run [4]:
run [5]:
run [6]:
run [7]:
run [8]:
run [9]:
run [10]:
run [11]:
run [12]:
run [13]:
run [14]:
run [15]:
run [16]:
run [17]:
run [18]:
run [19]:
run [20]:
1 Like
#2

That is how the lbfgs algorithm works, it evaluates the function at few points, here evaluating the function corresponds to calling the closure function.

1 Like
#3

Thanks for your answer. @albanD

1 Like