Issue with evaluators on binary dataset at higher epoch iterations

Hi.

I am new to PyTorch (am learning it in Google Colab) and am having an issue I can’t resolve. I am working with the BreastMNIST data set and am processing it as binary data.

All worked fine until I reworked it to use Ignite for the metrics. On a handful of epoch iterations all works ok. However, when I increase the epochs to 10+ the data training works fine but the evaluation throws a “For binary cases, y_pred must be comprised of 0’s and 1’s” error.

Any thoughts? I’m particularly confused given the exact same dataset is handled just fine when running with fewer epochs.

Thanks.

Error
ERROR:ignite.engine.engine.Engine:Current run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's.
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-27-950c571740af> in <cell line: 1>()
----> 1 train_evaluator.run(train_loader_at_eval)
      2 metrics = train_evaluator.state.metrics
      3 print(f"Training Eval Results: AUC: {metrics['auc']:.2f}   Accuracy: {metrics['accuracy']:.2f}")
      4 
      5 val_evaluator.run(val_loader)

/usr/local/lib/python3.10/dist-packages/ignite/metrics/accuracy.py in _check_binary_multilabel_cases(self, output)
     56 
     57         if not torch.equal(y_pred, y_pred**2):
---> 58             raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
     59 
     60     def _check_type(self, output: Sequence[torch.Tensor]) -> None:

ValueError: For binary cases, y_pred must be comprised of 0's and 1's

Hi @katyapanf

However, when I increase the epochs to 10+ the data training works fine but the evaluation throws a “For binary cases, y_pred must be comprised of 0’s and 1’s” error.

Accuracy, Precision, Recall metrics for binary case require binarized predictions:

def binarize_preds(output):
    y_preds, y_true = output
    return y_preds > 0.5, y_true


precision = Precision(average=False, output_transform=binarize_preds)
recall = Recall(average=False, output_transform=binarize_preds)
accuracy = Accuracy(output_transform=binarize_preds)

Any thoughts? I’m particularly confused given the exact same dataset is handled just fine when running with fewer epochs.

In your original code, evaluation part is running in the end of the training only. Usually, we may want to run model evaluation during the training, for example, every epoch ended. Using ignite, this can be done like this:

trainer = Engine(training_step)
train_evaluator = Engine(eval_step)
val_evaluator = Engine(eval_step)

# Schedule when to run model validation:
@trainer.on(Events.EPOCH_COMPLETED)
def run_validation():
    def log_metrics(epoch, tag, metrics):
        metrics_output = "\t".join([f"\t{k}: {v}" for k, v in metrics.items()])
        print(f"Epoch[{epoch}] - {tag} metrics: {metrics_output}")

    train_evaluator.run(train_loader_at_eval)
    metrics = train_evaluator.state.metrics
    log_metrics(trainer.state.epoch, "Training", metrics)

    val_evaluator.run(val_loader)
    metrics = val_evaluator.state.metrics
    log_metrics(trainer.state.epoch, "Validation", metrics)

trainer.run(train_loader, max_epochs=NUM_EPOCHS)

(For a better understanding of how events system is working, you can try this playground)

Here is a full training code with few updates in colab:

Hope this helps!

That did the trick, thank you!!