How to make ignite trainer handle error batch instead of stop the program?

ignite
1

Sometime, the single sample that does not work with model (dimension mismake) or None in batch (because error in loading data due to corrupt).

Instead of throwing exception and stop the program, how to just do handle (log, write something) and carry-on with Ignite trainer.

What should I do ? If possible, please give example in code.

2

@ThaiThien currently there is no proper way to skip a single iteration (Maybe it could worth a feature request).
However, still what can be done is the following:


def process_function(engine, batch):
     if not is_correct_batch(batch):
         # let's return the previous output => wont break other handlers using state.output
         # but metrics computation will be impacted... 
         return engine.state.output

     # otherwise, continue as if everyhing is OK
        
engine = Engine(process_function)

Hope this could help with your problem.

PS: feel free to open a feature request on the github if a proper handling of such situation is needed for you (and maybe others). Thanks