Skip to content

Add regression tests for numeric-distribution issues #166, #209, #210, #212 - #389

Open
scks1 wants to merge 1 commit into
Metaculus:mainfrom
scks1:add-numeric-report-issue-regression-tests
Open

scks1 wants to merge 1 commit into
Metaculus:mainfrom
scks1:add-numeric-report-issue-regression-tests

Conversation

@scks1

@scks1 scks1 commented Oct 31, 2026

Copy link
Copy Markdown

Adds test_numeric_report_issue_regressions.py: 14 regression tests for NumericDistribution anchored to four open issues. Happy to fold these into test_numeric_report.py instead if you prefer one file. Every test in this PR passes on current master.

#166 — one test locks the documented behaviour for well-separated repeated values: they stay sorted and are moved by at most the 1e-6 nudge. Two further characterization tests for the in-bounds branch of _check_and_update_repeating_values currently fail on master and are deliberately not included, so this PR keeps the suite green:

  • Declaring 9.99999995 @0.85, 10 @0.9, 10 @0.95 (bounds 0–100) validates to values [9.99999995, 9.9999999, 9.99999995, …] — no longer increasing: the downward nudge crosses a distinct neighbour, and the result is never re-checked after the repair.
  • Declaring repeats at 0.0000005 against a closed lower bound of 0 validates to [-4e-07, -3e-07, …] — the nudge moves in-bounds values below the closed bound.

Those two tests assert the invariants (strictly increasing declared values; in-bounds inputs stay within closed bounds) rather than any particular fix. Happy to add them alongside whatever robust solution #166 lands on, or to share them now if they are useful.

#212 — extends the existing repeated-value coverage to the 201-point CDF where 200 bins fit the 0–20 range exactly (mass lands in the bucket ending at 12.0), and locks that concentrated forecasts and aggregation of identical near-bound forecasters produce strictly increasing CDFs (problem 2's workflow error does not reproduce on current code).

#209 — first tests for _check_percentile_spacing: declared heights closer than 5e-05 are rejected, and generated CDFs respect the same floor. (The original report points at a Discord thread, so these lock the spacing behaviour the title describes rather than claiming to close the issue outright.)

#210 — for smooth closed-bound and log-scaled forecasts the PMF shows no edge elevation; where a forecast genuinely declares heavy near-bound mass, standardization caps every bin at the documented maximum.

Test run (Python 3.12, repo lockfile): new file — 14 passed; existing test_numeric_report.py unchanged at 147 passed / 96 skipped, and unaffected when run together (161 passed, 96 skipped).

Disclosure: this contribution was written by Árbol, an AI assistant working under the direction of the account holder, who set the acceptance criteria and verified the results before submission.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant