Calibration

[heavy leaf]

I am not ale to understand the concept of calibration of a ML model. Read an article on this and it mentions about adjusting the output probabilities of a model to be closer to actual empirical probabilities. https://medium.com/analytics-vidhya/calibration-in-machine-learning-e7972ac93555

But is it not like, you can create a bad model and then use calibration to make it good? Also, does calibration apply to regression model as we are already using MSE/log loss to predict the probabilities?
I am so confused, can you please provide some insight?

[vernal lodge]

This may help a bit more: https://machinelearningmastery.com/probability-calibration-for-imbalanced-classification/

But basically, it's not saying that the model itself is bad. You don't "retrain" a model or anything. What's really being calibrated is not the model itself, but the output of model.

While models output a probability-like output, they don't necessarily mean actual probability, or the "given X, likelihood of Y". Most models are not probabilistic, they don't learn a prior.

Say that based on probability, there should be 5% of true class, in other words, 5% are fraud. We know this by analyzing the past data distribution. It's a classic imbalanced dataset. You train your model, and you likely balanced the dataset so that your model can learn the minority class, this is commonly done in model training.

But now solely based on the output of your model, your model is predicting around 20% of the samples are fraud. It's a combination of threshold selection and such, but basically if you only apply threshold on your model output, it's not aligned with actual probability of fraud.

Therefore, you apply some scaling or transformation on top of model output, the logits, so that the calibrated output/logits actually are more aligned with actual probability, say 4-7%, rather than 20% which is way off.

It's just to ensure that your model is not too off from the actual probability overall