When is the closure called by the optimizer (in the optional and required cases)?

1

On the Pytorch documentation https://pytorch.org/docs/stable/optim.html, the code to call .step() for an optimizer that needs a closure is:

for input, target in dataset:
    def closure():
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        return loss
    optimizer.step(closure)

I have two similar questions:

  • If I use an optimizer that needs a closure, would the closure function be called for the first evaluation ? Because the call to zero_grad(), backward()… does not appear outside the closure in the snippet.
  • If I use an optimizer that does not needs a closure and I provide one, do I still need to manually call zero_grad(), backward() before calling .step(closure) ?

Those two question sum up as: if I provide a closure function, will it be called no matter the optimizer, therefore in all cases I do not need to call zero_grad(), backward()… before calling .step(closure) ?