Skip to content

Reject classification models in conformal models - #3282

Open
shivenaggarwal wants to merge 2 commits into
unit8co:masterfrom
shivenaggarwal:fix/conformal-reject-classifiers
Open

shivenaggarwal wants to merge 2 commits into
unit8co:masterfrom
shivenaggarwal:fix/conformal-reject-classifiers

Conversation

@shivenaggarwal

Copy link
Copy Markdown

Checklist before merging this PR:

  • Mentioned all issues that this PR fixes or addresses.
  • Summarized the updates of this PR under Summary.
  • Added an entry under Unreleased in the Changelog.

Related to #2884.

Summary

ConformalNaiveModel and ConformalQRModel currently accept fitted classification models (SKLearnClassifierModel, LightGBMClassifierModel, XGBClassifierModel, CatBoostClassifierModel) without any error. The resulting intervals are built around class labels, so they don't mean anything. For example, with class labels {0, 1, 2} the calibrated upper quantiles come out as 3 or 4.

As mentioned in #2884:

Our conformal models should only be used with the regression models (e.g. models predicting a continuous numerical value).

This adds a check to ConformalModel.__init__, so it applies to both conformal models:

ValueError: `model` must be a regression model (predicting continuous numerical values). Classification models are not supported.

Classifiers are detected with the existing _model_type property (ModelType.FORECASTING_CLASSIFIER, set by _ClassifierMixin). Models that don't define it (torch, foundation, ensemble models, ...) are not affected, and the conformal module doesn't need to import sklearn_model.

The sentence "Conformal models can be applied to any of Darts' global forecasting model..." in the three conformal docstrings now also states that classification models are not supported.

This PR only adds the guard. It does not implement conformal prediction for classifiers, which is what #2884 requests, so it should not close the issue.

Other Information

  • New test test_model_construction_classifier, parametrized over both conformal models. It checks that a fitted SKLearnClassifierModel is rejected with the message above and that a regression model fitted on the same series is still accepted. On master it fails with DID NOT RAISE ValueError.
  • pytest darts/tests/models/forecasting/test_conformal_model.py: 740 passed. The conformal tests in the serialization, probabilistic and historical forecasts modules also pass.
  • I also checked by hand that CatBoostClassifierModel is rejected and CatBoostModel is still accepted.
  • pre-commit run passes on the changed files.

Conformal models only support regression models (forecasting continuous
numerical values). Classification models such as `SKLearnClassifierModel`
were silently accepted, which resulted in meaningless intervals around
the predicted class labels. `ConformalModel` now raises a `ValueError`
for classification models.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant