Fix skipped BCal aggregation and retired judge model

[martinsrui-msft]

Scheduled BCal runs skipped final aggregation when a dataset entry run failed. That left no completeness summary and made evaluation data appear to disappear. NL2AL scoring also used the retired GPT-4.1 ShortCo judge.

• Always aggregate artifacts and report expected, produced, missing, unexpected, and duplicate results.
• Preserve diagnostics, then fail incomplete runs before scoring, external storage, or leaderboard publication.
• Isolate artifacts by rerun attempt and switch LMChecklist/NL2AL scoring to gpt-56-reasoning-nano-luna with separate judge baselines.
• Pin and cache BcContainerHelper once per workflow.

Verification

Check	Result
Focused tests	130 passed
Full test suite	1,094 passed, 1 skipped, 1 deselected
Pre-commit	Passed
Hosted PR CI	17/17 passed

[github-code-quality]

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest

The overall line coverage in commit 59bc801 in the martinsrui-harden-bc... branch is 85%. The line coverage in commit 973eb05 in the main branch is 86%.

Show a line coverage summary of the most impacted files.

File	main 973eb05	martinsrui-harden-bc... 59bc801	+/-
src/bcbench/eva...st_implement.py	65%	53%	-12%
src/bcbench/evaluate/bugfix.py	47%	36%	-11%
src/bcbench/eva...stgeneration.py	43%	37%	-6%
src/bcbench/commands/result.py	93%	89%	-4%
src/bcbench/results/display.py	93%	92%	-1%
src/bcbench/com...ands/dataset.py	25%	26%	+1%
src/bcbench/exceptions.py	71%	74%	+3%
src/bcbench/ope...c_operations.py	0%	72%	+72%
src/bcbench/res...completeness.py	0%	96%	+96%
src/bcbench/ope...t_operations.py	0%	97%	+97%

---

_{Updated October 06, 2026 12:25 UTC}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 · 3

Open (4)

• This uses: value is not a valid path for a reusable workflow invocation. GitHub Actions expects a… · New
• Writing the raw exception text to stderr can leak sensitive or high-volume information into build… · New
• The model field is named complete, but exported per-row metadata uses evaluation_complete. This… · New
• The model field is named complete, but exported per-row metadata uses evaluation_complete. This… · New

What changed in this PR

Hardens scheduled BCal evaluation ingestion by adding matrix completeness tracking, ensuring aggregation/diagnostics are preserved across shard failures, and preventing scoring/storage/leaderboard updates for partial runs.

Changes:

• Add completeness metadata generation/export (JSON + per-result metadata) and a result require-complete CLI gate.
• Update GitHub Actions workflows to always summarize after failures, isolate artifacts per run_attempt, and block bceval/leaderboard unless complete.
• Pin/cache BcContainerHelper and restore narrowly-scoped CAPI retry logic; update LMChecklist judge model pinning and aggregation identity.

File	Description
tests/​test_review_workflows.py	Adds workflow regression tests for completeness enforcement and pinned dependency caching.
tests/​test_evaluation_summary.py	Updates aggregation tests to keep retired judge baseline runs separated.
tests/​test_completeness.py	Introduces unit tests for completeness counting and metadata export.
tests/​test_cli_commands.py	Adds integration coverage for completeness persistence and step outputs.
tests/​test_category_command.py	Updates expected pinned judge model in generated bceval config.
tests/​test_bcal_capi_bridge.py	Adds tests asserting retry only for transient CAPI failures.
src/​bcbench/​results/​summary.py	Extends summaries to optionally carry completeness metadata.
src/​bcbench/​results/​display.py	Renders completeness in GitHub summaries and adds a completeness-only summary writer.
src/​bcbench/​results/​completeness.py	Implements completeness model, persistence, metadata export, and markdown rendering.
src/​bcbench/​results/​bceval_export.py	Embeds completeness metadata into each exported bceval JSONL row.
src/​bcbench/​results/​__init__.py	Re-exports completeness and the new GitHub completeness summary helper.
src/​bcbench/​commands/​result.py	Adds completeness tracking to result summarize and a require-complete gate command.
src/​bcbench/​commands/​dataset.py	Adds --count-output for reusable workflows to retrieve entry counts.
src/​bcbench/​agent/​shared/​config.yaml	Pins LMChecklist judge model to gpt-56-reasoning-nano-luna.
src/​bcbench/​agent/​bcal/​bc_eval_capi_bridge.py	Restores bounded retry for transient CAPI failures with backoff.
CATEGORIES.md	Documents pinned judge model and aggregation separation from retired baselines.
.github/​workflows/​summarize-results.yml	Always uploads partial diagnostics; blocks scoring/leaderboard on incompleteness; enforces completeness at end.
.github/​workflows/​get-entries.yml	Exposes entry count output for downstream completeness enforcement.
.github/​workflows/​bcal-evaluation.yml	Adds pinned/cached BcContainerHelper setup, run_attempt-scoped artifacts, and always-run summarization.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[martinsrui-msft]

Superseded by the replacement native stack:

• Fix Luna LM Checklist request formatting #943 — LM Checklist judge migration and required Luna compatibility
• Fix skipped BCal aggregation #944 — BCal aggregation/completeness and workflow hardening, with no execution retries

Together, the two PRs preserve this PR's cumulative patch while separating the concerns for review.