Skip to content

Fix Luna LM Checklist request formatting - #943

Open
martinsrui-msft wants to merge 1 commit into
mainfrom
martinsrui-lm-checklist-model
Open

martinsrui-msft wants to merge 1 commit into
mainfrom
martinsrui-lm-checklist-model

Conversation

@martinsrui-msft

@martinsrui-msft martinsrui-msft commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • convert assistant LM Checklist history to Responses-compatible output_text content for GPT-5 routes
  • omit Luna's unsupported temperature argument without changing other models
  • cover Luna, non-Luna GPT-5 Responses, and legacy Chat Completions behavior

Context

main already selects gpt-56-reasoning-nano-luna through #946. In run 37611607449, all 110 LM Checklist calls failed before scoring because autoevals==0.3.0 converted assistant history to input_text; CAPI accepts assistant history only as output_text or refusal on the Responses API.

This PR adds a narrow request-side adapter around the built-in bc-eval LM Checklist scorer. It does not change checklist assertions, grading semantics, score names, retries, or timeouts. #948 addresses the separate response-side issue where nested Responses output is otherwise converted to content=None.

The generic role-aware conversion should ultimately live in autoevals, or in bc-eval as the integration boundary if that dependency cannot be updated immediately. This local adapter keeps the currently pinned bc-eval==0.3.14 workflow operational and can be removed after the upstream integration is fixed and upgraded.

Validation

  • 3 focused LM Checklist request-formatting tests passed
  • Ruff check and format checks passed
  • targeted Ty checks passed
  • relevant pre-commit hooks passed

@github-code-quality

github-code-quality Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: Python

Python / code-coverage/pytest

The overall line coverage in commit 7df241b in the martinsrui-lm-checkl... branch remains at 86%, unchanged from commit d19989f in the main branch.


Updated October 07, 2026 14:03 UTC

@haoranpb

Copy link
Copy Markdown
Collaborator

How about something like #946 ?

I see something is failing in pre-commit, will make a separate fix

@haoranpb Sun Haoran (haoranpb) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can probably close this and rebase #944 to main

@martinsrui-msft
martinsrui-msft force-pushed the martinsrui-lm-checklist-model branch from a18ea59 to a8a746f Compare October 7, 2026 13:52
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@martinsrui-msft
martinsrui-msft force-pushed the martinsrui-lm-checklist-model branch from a8a746f to 7df241b Compare October 7, 2026 14:01
@martinsrui-msft martinsrui-msft changed the title Switch LM Checklist judge to Luna Fix Luna LM Checklist request formatting Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants