You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Type agent config upstream to fix ty unsound-assignment - #952
Fix ty unsound-assignment by typing data where it is read, instead of annotating each place Any gets assigned.
config.yaml is parsed into a frozen pydantic AgentConfig (bcbench/types.py). The YAML shape is unchanged; each section is its own model (instructions/skills: ToggleConfig, agents: CustomAgentsConfig, mcp: McpConfig). Unknown keys are rejected at every level, toggles are StrictBool, and MCP servers are typed as either HttpMcpServer or StdioMcpServer; the reserved altool/bcmcp servers must be stdio/http.
mcp.py keeps its original flow, including updating the server entries in place. The changes are just dict lookups swapped for attribute access, plus isinstance narrowing in the two next(...) lookups.
JudgeConfig is now a pydantic model too. It reads judges.<name>.model with AliasPath, which replaces the hand-written checks and the cast.
Values from raw JSON keep explicit annotations: text: object and platform: object, narrowed with isinstance. One cast with a comment remains for re.Match.group(1), which typeshed types as str | Any. In evaluator/metrics.py, tool_usage is untyped bc-eval metadata, so it stays Any instead of being annotated as dict[str, int].
Behaviour changes
An invalid config now fails when the file is loaded, with a pydantic ValidationError. This covers test-generation input, MCP server type, <category>-template keys, judge models, misspelled or non-mapping sections, non-boolean toggles (e.g. enabled: "yes") and mistyped reserved MCP servers. The old "use hyphens" hint is gone, but the error still lists the valid values.
A missing instructions/skills/agents section now means disabled. Disabled plugins entries are still not validated.
Part of the stack that makes ty strict (all = "error") across the whole uv workspace. Each PR fixes one rule in Python code only; the config change lands in #956.
Sun Haoran (haoranpb)
changed the title
Make Any boundaries explicit for ty unsound-assignment
Type agent config upstream to fix ty unsound-assignment
Oct 7, 2026
Reject prompt.templates to prevent silent overwrite and later KeyError
src/bcbench/types.py:582
An explicit prompt.templates key is currently accepted but silently overwritten by the synthesized mapping. This bypasses the promised unknown-key validation, so the config loads successfully and build_prompt later fails with a KeyError for the missing category template. Reject this internal field in the pre-validator so YAML must keep using <category>-template keys.
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Parse config.yaml into a validated, frozen AgentConfig model instead of
passing raw dicts around, so consumers use typed attributes and ty no longer
sees Any flowing into declared types.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 70c54f79-d7a0-4cde-baf2-e53dfff3deb4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
What
Fix ty
unsound-assignmentby typing data where it is read, instead of annotating each placeAnygets assigned.config.yamlis parsed into a frozen pydanticAgentConfig(bcbench/types.py). The YAML shape is unchanged; each section is its own model (instructions/skills:ToggleConfig,agents:CustomAgentsConfig,mcp:McpConfig). Unknown keys are rejected at every level, toggles areStrictBool, and MCP servers are typed as eitherHttpMcpServerorStdioMcpServer; the reservedaltool/bcmcpservers must bestdio/http.mcp.pykeeps its original flow, including updating the server entries in place. The changes are just dict lookups swapped for attribute access, plusisinstancenarrowing in the twonext(...)lookups.JudgeConfigis now a pydantic model too. It readsjudges.<name>.modelwithAliasPath, which replaces the hand-written checks and thecast.text: objectandplatform: object, narrowed withisinstance. Onecastwith a comment remains forre.Match.group(1), which typeshed types asstr | Any. Inevaluator/metrics.py,tool_usageis untyped bc-eval metadata, so it staysAnyinstead of being annotated asdict[str, int].Behaviour changes
ValidationError. This covers test-generation input, MCP server type,<category>-templatekeys, judge models, misspelled or non-mapping sections, non-boolean toggles (e.g.enabled: "yes") and mistyped reserved MCP servers. The old "use hyphens" hint is gone, but the error still lists the valid values.instructions/skills/agentssection now means disabled. Disabledpluginsentries are still not validated.Part of the stack that makes ty strict (
all = "error") across the whole uv workspace. Each PR fixes one rule in Python code only; the config change lands in #956.