Average time per epoch calculation on pytorch ignite

Hello,

I need to save a file with the average elapsed time per epoch and I found this function on the documentation:

https://pytorch.org/ignite/_modules/ignite/handlers/timing.html

 t = Timer(average=True)
for _ in range(10):
       work()
       idle()
       t.step()
def save_time(engine):
     torch.save(t.value(),'time.pt´)

trainer.add_event_handler(Events.EPOCH_STARTED(every=3), save_time)

this works but It shows the elapsed time in 3 epochs and I suppose the file overwrites every 3 epochs but what I want is to calculate the average time just once for the first epoch and stop the timer. I don’t want to divide timer value between 3, also I don’t want to generate one file per epoch. One option is to run my code with just one epoch but I am running several models and I would like to generate this just once every time I am going to train.

How can I achieve this? :smiley:

pd: the documentation show an example doing it by iteration, but I am unable to translate this example to epoch. I believe the documentation should extend to epoch example as well.

Regards,

@Tanya_Boone Thank you for this question.

I think that several things could help you.

First, you can use Timer with filtered event. I think that in your case, you should try the following code to reset the timer every 3 epochs (started)

timer.attach(trainer,
             start=Events.EPOCH_STARTED(every=3),
             resume=Events.ITERATION_STARTED,
             pause=Events.ITERATION_COMPLETED)

An other option is to use directly Timer (without step to do not average)

timer = Timer()

@trainer.on(Events.STARTED)
def time_start():
    # reset the timer at the beginning
    timer.reset()

@trainer.on(Events.EPOCH_STARTED(every=3))
def time_start():
    # use value() to get the time
    print("timer of 3 epochs :", timer.value())
    # reset the timer from now for 3 next epochs
    timer.reset()

Last point I would like to mention, we recently added epoch timers in trainer.state.timers. That is another option which should help you to mesure epoch times

@trainer.on(Events.EPOCH_COMPLETED)
def print_timers():
    # trainer.state.timers is a dict
    print("timers=", trainer.state.timers)

If you need more support, don’t hesitate to ask, it would be a pleasure to help :slight_smile:

HTH

Thanks Sylvain !

@Tanya_Boone please let us know if Sylvain’s answer solves your problem. Thanks !

Hello,

I am trying with this method

timer.attach(trainer,start=Events.EPOCH_STARTED(every=1),
    resume=Events.ITERATION_STARTED,
    pause=Events.ITERATION_COMPLETED)

After I want to print the value

print(“timers=”, trainer.state.timers)

Engine run is terminating due to exception: ‘State’ object has no attribute 'timers". I changed to timer but I am getting the same error :frowning:

This feature is not in stable 0.3.0. You have to use git version or nightly version. Sorry to miss that point.

To install nightly release: https://github.com/pytorch/ignite#nightly-releases

1 Like