Pytorch model calibration

1

Hi,
I tried to calibrate my pytorch model with sklearn.
I used the following code

valid_X, valid_y = get_input_and_target_from_loader(valid_loader)
# calibrate model on validation data
calibrator = CalibratedClassifierCV(model, cv='prefit', method='isotonic')
calibrator.fit(valid_X, valid_y)

I get an error:
sklearn.utils._param_validation.InvalidParameterError: The 'estimator' parameter of CalibratedClassifierCV must be an object implementing 'fit' and 'predict_proba', an object implementing 'fit' and 'decision_function' or None.

Is there any equivalent function for Pytorch models ?

2

I’m not familiar with sklearn + calibration, so I have a dumb question: what is the difference between calibrating a model and training a model (e.g. forward, backward, optimizer step update parameters), or do you mean something closer to optimizing hyper parameters?

3

I didn’t mean to optimize hyper-parameters. Here is a link explaining calibration.

2 Likes
4

If I’m not mistaken, in order to use sklearn’s CalibratedClassifierCV, you need to pass it a sklearn Model in the model argument.
This won’t work out of the box with PyTorch models.

However, the link you shared in your last post is very illustrative and shares some examples to implement your own calibration using sklearn’s IsotonicRegression model.