| Documentation | Build Status | Code Quality | License & DOI | Downloads |
|---|---|---|---|---|
  |
   |
   |
 |
  |
| ForwardDiff | ReverseDiff (tape) | Enzyme forward | Enzyme reverse | Mooncake reverse | Mooncake forward |
|---|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
Proper scoring rules for evaluating probabilistic forecasts in Julia.
ScoringRules.jl is a port of the R package
scoringRules by Alexander I. Jordan,
Fabian Krüger, Sebastian Lerch and Sam Allen. Its formulae and algorithms follow
that package, whose methods are rooted in the scientific literature on
probabilistic forecasting. The package is expected to diverge from upstream
over time. If you use it, please cite the
original authors (see Attribution and provenance).
It is distributed under the same licence as the original, GPL-2.0-or-later.
The initial port was generated by a large language model (Claude Opus 4.8) under human guidance, translating the R and C++ source into idiomatic Julia. It introduced no statistical methodology of its own.
The formulae and algorithms follow the R package; the interface is Julia's. What that means in practice:
- Forecasts are Distributions.jl objects, dispatched on their type, instead of a
separate function per family: write
crps(Normal(0, 1), 0.4), notcrps_norm. - Gradients come from automatic differentiation with any supported backend, so
there is no equivalent of R's hand-written
gradcrps_*/hesscrps_*. - Results match R throughout (the test suite checks against reference values
generated from R; see
Validation against R
for a live value- and gradient-level comparison). The one genuine numerical
difference is the log-logistic Dawid–Sebastiani score, where R has a bug that
returns
NaNand we return the correct value; a couple of other choices differ in method but not in results (e.g. an analytic GEV Gumbel limit in place of R's numerical integration). Both are documented under Differences from R. LogLaplace,TwoPieceNormalandTwoPieceExponentialare provided here, as Distributions.jl does not have them.
- Univariate scores — CRPS (
crps), the logarithmic score (logs) and the Dawid–Sebastiani score (dss), for parametric (Distributions.jl) and ensemble forecasts alike. - Multivariate scores — energy score (
es), variogram score (vs) and the maximum-mean-discrepancy score (mmds) for ensemble forecasts. - Quantile and categorical scores —
quantile_score,interval_scoreand the ranked probability score (rps).
All scores are negatively oriented, so a lower score is a better forecast, and they broadcast with the usual dot syntax.
using ScoringRules, Distributions
crps(Normal(0, 1), 0.4) # parametric forecast
logs(Poisson(3), 2) # log score of a count forecast
crps(randn(1000), 0.4) # ensemble (sample) forecastSee the documentation for the full list of supported families and scores.
- scoringRules (R) — the reference implementation this package ports; the source of the parametric, sample and weighted scoring rules provided here, and of the values the test suite validates against.
- scoringutils (R) — applied
forecast-evaluation utilities built on top of
scoringRules. - Distributions.jl — supplies the distribution types that the parametric scores dispatch on.
- CensoredDistributions.jl — sibling EpiAware package whose structure, testing and CI this package follows.
- PostForecasts.jl and ForecastBaselines.jl — other Julia packages that touch probabilistic-forecast scoring.
ScoringRules.jl began as a Julia port of the R scoringRules package (version
1.1.3), whose methods are rooted in the scientific literature on probabilistic
forecasting. That package was written by:
- Alexander I. Jordan (0000-0001-7423-1352)
- Fabian Krüger (0000-0002-5112-9037)
- Sebastian Lerch (0000-0002-3467-4375)
- Sam Allen (0000-0003-1971-8277)
- Maximiliane Graeter (contributor)
Please cite their work:
Jordan, A., Krüger, F., & Lerch, S. (2019). Evaluating Probabilistic Forecasts with scoringRules. Journal of Statistical Software, 90(12), 1–37. https://doi.org/10.18637/jss.v090.i12
For the weighted scores (twcrps, owcrps, and relatives):
Allen, S. (2024). Weighted scoringRules: Emphasizing Particular Outcomes When Evaluating Probabilistic Forecasts. Journal of Statistical Software, 110(8), 1–26. https://doi.org/10.18637/jss.v110.i08
The initial LLM-generated port introduced no statistical methodology of its own.
The package will grow features the R package does not have; the citation request
above covers everything that traces back to scoringRules. Any errors from
translation or later development are ours, not the original authors'; please
report discrepancies with the R package as bugs here.
We welcome contributions and new contributors! Please open an issue or pull request on GitHub. This package follows ColPrac and the SciML style.
If you use ScoringRules in your work, please cite it. Citation metadata lives in CITATION.cff, which GitHub renders as a "Cite this repository" button on the repository page.
Please note that the ScoringRules project is released with a Contributor Code of Conduct. By contributing, you agree to abide by its terms.