Skip to content

Fix Luna LM Checklist response parsing - #948

Closed
martinsrui-msft wants to merge 1 commit into
martinsrui-bcal-resilient-aggregationfrom
martinsrui-fix-luna-response-parsing
Closed

martinsrui-msft wants to merge 1 commit into
martinsrui-bcal-resilient-aggregationfrom
martinsrui-fix-luna-response-parsing

Conversation

@martinsrui-msft

@martinsrui-msft martinsrui-msft commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • fix Luna LM Checklist scoring by extracting nested Responses API assistant output_text before autoevals converts it to content=None
  • scope the compatibility client to the exact Luna path while delegating non-Luna requests unchanged
  • preserve supported Responses parameters and raise an explicit error when no output text is returned

Failure reproduced

In run 37596204440, job 112734686674, all 110 LM Checklist calls failed when bc-eval 0.3.14/autoevals 0.3.0 converted the nested CAPI response into choices[0].message.content=None. The built-in scorer then called json.loads(None). Kusto's missing test_passed error was downstream because no row received any scorer output.

This PR is the new top layer above #944.

Validation

  • 6 focused pytest cases passed
  • Ruff check and format passed
  • full Ty check passed
  • relevant pre-commit hooks passed
  • local integration with cached bc-eval 0.3.14/autoevals 0.3.0 reproduced the original content=None behavior and then produced test_passed=1.0 through the real scorer
  • real BCal test run 37613146969 passed on all 4 entries, logged zero LM Checklist exceptions, produced the test_passed score, and completed Braintrust/Kusto storage

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@martinsrui-msft
martinsrui-msft added this pull request to stack #945 October 7, 2026 11:17
@github-code-quality

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest

The overall line coverage in commit 55a8f41 in the martinsrui-fix-luna-... branch remains at 86%, unchanged from commit d19989f in the martinsrui-bcal-resi... branch.

Show a line coverage summary of the most impacted files.
File martinsrui-bcal-resi... d19989f martinsrui-fix-luna-... 55a8f41 +/-
src/bcbench/commands/result.py 93% 89% -4%
src/bcbench/results/display.py 93% 92% -1%
src/bcbench/results/summary.py 96% 96% 0%
src/bcbench/res...lts/__init__.py 100% 100% 0%
src/bcbench/res...ceval_export.py 100% 100% 0%
src/bcbench/com...ands/dataset.py 25% 26% +1%
src/bcbench/res...completeness.py 0% 96% +96%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant