[mohsenhariri]

Add optional binary Bayes@N posterior summaries through three registered task names: humaneval-bayes-at-n, humaneval-unstripped-bayes-at-n, and mbpp-bayes-at-n. They retain the original prompts, references, postprocessing and program checks, and add bayes_at_n (posterior mean) and bayes_at_n_std (posterior SD) alongside pass@k. Existing task names and output dictionaries are unchanged.

accelerate launch main.py --tasks humaneval-bayes-at-n \
    --load_generations_path generations.json --n_samples 16 \
    --allow_code_execution --metric_output_path evaluation_results.json

The native standard-library helper implements the binary, uniform-prior case of Bayes@N, Algorithm 1 and Appendix A. Each problem has an independent Beta(1,1) prior; corpus uncertainty is sqrt(sum(problem posterior variances))/problem_count. No runtime dependency is added. Both fields use the [0,1] scale and describe the fixed problem set's average latent correctness, separately from pass@k.

The metric checks complete equal-size program rows, matching test references, expected sample counts and unique binary execution verdicts. It rejects intermediate checkpoints and nonzero offsets with loaded generations: the existing resume/loader paths can otherwise shift question identities. Invalid or empty loaded files raise a clear error before execution. Historical observations, categorical rubrics and credible intervals are outside this scope.

Validation in a dedicated Python 3.9.25 environment, matching the CI runtime:

• 43 focused tests passed, including exact Fraction cases, a 164-question/64-trial workload, actual trusted program grading, task registration, parser/result JSON, unchanged original tasks and invalid/checkpoint cases.
• 24 seeded workloads matched independent rational moments and the pinned Scorio reference within 1e-14.
• Independent review found two data-path issues (checkpoint row compaction and empty-file indexing); both were fixed and covered by regression tests. The reviewer reran all 43 focused tests successfully.
• New Python files pass Black with Python 3.7 target grammar; changed Python files parse under the declared 3.7 grammar. git diff --check passes.

The full suite passed 40 tests before the final checkpoint/file-validation regressions were added. On the final source, an online rerun passed 45 tests and failed one existing test when a public dataset request returned HTTP 429. An offline rerun passed 44 and failed the two existing dataset-dependent tests; unchanged upstream reproduced both failures in the same environment. All 43 new tests pass independently. The legacy downloader also needed its public tiny-model cache populated because it cannot handle the Hub's relative redirects; no project or installed library code was patched.

I am a coauthor of the Bayes@N paper. Code, tests and documentation were prepared with Codex assistance.

[mohsenhariri]

The CI run for 7028bda is waiting for maintainer approval, with no jobs executed. Could a maintainer approve it and review the separate HumanEval/MBPP Bayes@N task variants? The PR body records the focused local tests and dataset-download limitations.