Pytorch model calibration

1

Hi,
I tried to calibrate my pytorch model with sklearn.
I used the following code

valid_X, valid_y = get_input_and_target_from_loader(valid_loader)
# calibrate model on validation data
calibrator = CalibratedClassifierCV(model, cv='prefit', method='isotonic')
calibrator.fit(valid_X, valid_y)

I get an error:
sklearn.utils._param_validation.InvalidParameterError: The 'estimator' parameter of CalibratedClassifierCV must be an object implementing 'fit' and 'predict_proba', an object implementing 'fit' and 'decision_function' or None.

Is there any equivalent function for Pytorch models ?

2

I’m not familiar with sklearn + calibration, so I have a dumb question: what is the difference between calibrating a model and training a model (e.g. forward, backward, optimizer step update parameters), or do you mean something closer to optimizing hyper parameters?

3

I didn’t mean to optimize hyper-parameters. Here is a link explaining calibration.

1 Like