[AzulGarza]

Summary

• Adds ConformalIntervals and optional prediction_intervals on model forecasters so forecast() can build level / quantiles from per-series cross-validation absolute errors on the point forecast (replacing native quantiles when enabled).
• Documents usage in docs/conformal-prediction.md, a electricity-price notebook, and forecasting-parameters; exports ConformalIntervals from the package root.
• Tests: unit tests for interval math and stub-forecaster integration; parametrized conformal_models contract test (Chronos bolt tiny + Tafsut) marked @pytest.mark.models.

Test plan

• uv run pytest tests/core/test_conformal.py tests/core/test_conformal_forecaster.py -n 0
• uv run pre-commit run on touched files
• CI test job (includes @pytest.mark.models conformal contract test with HF cache)

Made with Cursor

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[codspeed]

Unable to generate the performance report

Your organization has exceeded its Macro Runners budget. Because this run uses tokenless mode, CodSpeed needs to wait for the GitHub Actions workflow to complete to validate the upload. Since the walltime jobs will never run, the workflow stays pending and we are unable to process this run.

To run these benchmarks again, update your plan or increase your Macro Runners budget.

If you need help, please contact us at support@codspeed.io or on Discord.

[AzulGarza]

Review

Fixed in this push

• Calibration leaked the full panel (aed636e). FoundationForecast.forecast() builds panel from the full df and the calibration loop forwarded it to every window forecast. Models that read panel (Chronos, Tafsut, every TimeSeriesDataset.from_df user) then forecast all series from the full history: multi-series panels crashed with a broadcast error, and single series silently got residuals computed with future data. Calibration windows now always pass panel=None and only the window's own exog as X_df.
• Integer unique_id dropped intervals (aed636e). Scores were keyed by str(uid) but looked up by the raw value, so interval columns were silently missing.
• Regression tests for both (c37f1a9); they fail on the previous commit.
• test docs failure: docs/conformal-prediction.md called Tafsut(prediction_intervals=...) and used an undefined df. The snippet is now self-contained and uses attribute assignment (f6b1031).
• Notebook rewritten in the style of the other examples and executed (f6b1031).

Open

1. Model constructors don't accept prediction_intervals. Only Forecaster.__init__ takes it; none of the model __init__s forward it, so Tafsut(prediction_intervals=...) raises TypeError. Docs and notebook use model.prediction_intervals = ConformalIntervals(...) for now. Suggest adding the kwarg to every model (and GluonTSForecaster), plus a parametrized test over models that constructs with it.
2. Calibration doesn't batch. It runs one forecast per series per window, so a panel costs n_series × n_windows model calls with batch size 1. In the notebook (5 series, n_windows=7, 2 conformal models) that is ~70 extra calls per forecast() and ~1,300 tqdm bars. Calling self.cross_validation(df, h=h, n_windows=n_windows, step_size=h) once and grouping |y - yhat| by unique_id would batch across series and reuse the existing exog-aware CV path.
3. effective_n_windows capping is dead code. validate_conformal_samples already requires n_windows * h + 1 rows, so the cap never applies, while the docs say windows are capped for short series. Pick one behavior.
4. ConformalIntervals.h is public but always overwritten from forecast(h=...). Consider dropping it from the dataclass. method could be a Literal["conformal_distribution", "conformal_error"].
5. Interval column order differs from native output. Conformal emits lo-90, lo-80, hi-80, hi-90; native emits lo-80, hi-80, lo-90, hi-90 (visible in the notebook's holdout_df.head()).
6. cross_validation(level=...) on a conformal model silently returns native intervals. Worth a warning, or at least a note in the prediction_intervals docstring.
7. Test gaps: no end-to-end test for quantiles= (median handling), method="conformal_error" through forecast(), or exog calibration (the horizon_x path). test_native_level_without_prediction_intervals only asserts that the stub ignores level, so it doesn't exercise the library.

Notebook result (4 held-out days × 5 markets)

model	cov@80	cov@90	MASE
Tafsut	0.715	0.715	0.648
Tafsut-conformal	0.750	0.833	0.648
Chronos-2	0.729	0.865	0.677
Chronos-2-conformal	0.719	0.815	0.677

Tafsut's native 90% interval is clamped to its 0.1/0.9 knots, which makes it a good showcase. Chronos-2 doesn't benefit on this data.

CodSpeed Macro Runners and Entire Gates failures are external (billing / third-party) and unrelated to this PR.

[AzulGarza]

Follow-up:

• Open item 1 resolved (2c5c256): every model constructor now accepts prediction_intervals=ConformalIntervals(...), including the TimesFM dispatcher and GluonTS/Moirai. A test constructs each exported model with it.
• Docs (daeec03): enabled pymdownx.arithmatex + MathJax, so formulas render on the md page and in converted notebooks. Added a How it works section with the calibration-error and interval formulas.
• Notebook: now shows the case where conformal intervals clearly win. Tafsut and Chronos-Bolt predict only the 0.1–0.9 quantiles, so their native 95% interval equals the 80% one. On 7 held-out days × 5 markets:

model	cov@80	cov@95
Tafsut	0.745	0.745
Tafsut-conformal	0.802	0.937
Chronos-Bolt	0.777	0.777
Chronos-Bolt-conformal	0.839	0.943

[AzulGarza]

Correction to review item 6: cross_validation() does apply conformal intervals. It calls the model's forecast() on each training slice, so every window is recalibrated on data before its cutoff (the same design statsforecast uses). Added a test that locks this in (test_cross_validation_recalibrates_conformal_intervals_per_window), removed the wrong caveat from the docs, and replaced the notebook's manual holdout loop with ff.cross_validation(df, h=24, n_windows=7, level=[80, 95]), which produces identical metrics.

[codecov]

Codecov Report

❌ Patch coverage is 93.75000% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
foundationforecast/core/conformal.py	92.23%	8 Missing ⚠️
foundationforecast/core/forecaster.py	95.83%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!