How to load and save multiple schedulers?

1

Hello,

I am currently working on a project for computer vision to implement papers. I want to generalize the training script so that we can use multiple schedulers if we want.

I had 2 questions -

  • Is it absolutely necessary to save schedulers?
  • How to save/load multiple schedulers?

The workflow goes something like this -

train.yml

schedulers:
  # Scheduler1
  - name: "reduce_lr"
    mode: "min"
    factor: 0.1
    patience: 5
    verbose: True
  # Scheduler2
  # ..
  # Scheduler n

Loading scheduler(s) initially

def get_scheduler(optimizer, scheduler_params):
  scheduler_params["name"] in SCHEDULER_DICT:
  scheduler = scheduler_params["name"]
  del scheduler_params["name"]
  logging.info("Using '{}' scheduler for this run.".format(scheduler.upper()))

  scheduler = SCHEDULER_DICT[scheduler](optimizer=optimizer, **scheduler_params)
  return scheduler

sch_dict = {}  # scheduler(s)
    for scheduler_params in config.schedulers:
        sch_name = scheduler_params["name"]
        sch_dict[sch_name] = get_scheduler(optimizer=optimizer, scheduler_params=scheduler_params)

Calling schedulers

for name, scheduler in sch_dict.items():
                if name == "reduce_lr":
                    scheduler.step(val_metrics["loss"])
                else:
                    scheduler.step()