Skip to content

Judge: say how much of its scope each absence claim was given, and fetch references through MCP - #94

Merged
wu-sheng merged 1 commit into
mainfrom
judge-reading-mcp
Oct 28, 2026
Merged

wu-sheng merged 1 commit into
mainfrom
judge-reading-mcp

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Say beside each claim of absence how much of its scope the evaluator was given

Four claims of the judge say that something did not happen: a verdict of not_observed, the causes
question_not_asked and assumption_not_checked, and a person's goal found absent. The record cannot
prove any of them, so each stays the evaluator's judgment. A reader still needs to know how far to trust
one: whether the evaluator looked at the place where the thing would have shown.

Beside each claim, asz now reports how much of its scope the evaluator was given, counted in steps:

Count The steps
opened the evaluator asked for the step itself, and the answer was not empty
listed a row the evaluator was given named the step, and nothing more
none neither
unavailable among them all, the steps some of whose evidence no one can be given: a part cut, withheld, too large or dropped, or an image

The scope is the judged work for not_observed and absent, and for the two causes the span of one
stream from from, the step where the question or the assumption arose, to the finding's step. The
counts come from what each answer returned, counted only once it was given: for an API evaluator, when
a later request carried it to the model; for an agent CLI, when its navigation server wrote it to a log
that it closes with a count. A log that is not whole makes the counts unavailable, never smaller.

r3  final        an aside
    not observed, the evaluator's judgment; of the 40 steps of the judged work it opened 12,
    saw 21 only in lists and was not given 7; 2 hold evidence no one can be given

With it:

  • The two causes are renamed in plain words: clarification_not_sought is question_not_asked, and
    unsupported_assumption is assumption_not_checked.
  • A goal's first finding is its "first recorded" one, chosen by one function for the close and for
    -show, ties broken by the order the stages run.
  • A person's goal is absent only when every target was placed.
  • A tool call that shares a record with the agent's text reads as its arguments in asz.view, on the
    page and to the evaluators, never as that text.

Fetch references through an MCP server

A judge reference that needs a login, such as a Jira issue, can now be fetched by one tool of an MCP
server: a remote one over Streamable HTTP, or a local one asz starts over standard input and output.
asz speaks the protocol's revision 2026-07-28 and the four revisions before it that begin with a
handshake, and offers the server nothing.

judge:
  references:
    fetch:
      - prefix: https://example.atlassian.net/browse/
        mcp:
          url: https://mcp.atlassian.com/v2/mcp
          headers:
            Authorization: ASZ_JIRA_AUTHORIZATION
          tool: getJiraIssue
          arguments:
            cloudId: https://example.atlassian.net
            issueIdOrKey: "{rest}"

asz judge -reference-tools SRC lists the server's tools, to choose the tool and its arguments.

What asz keeps of a reference goes with the run when it is exported, so it keeps provenance and never a
credential:

  • A URL is kept without its query and fragment, and one with a user or a password stops the run before
    it writes anything.
  • An MCP server is kept as its URL's scheme and host, or its program's name; the arguments as sent, a
    secret one as [env NAME].
  • A failed fetch, over http, a command or MCP, is kept as a class and a detail asz builds, never a raw
    error that names the URL.
  • The secrets of a fetch are removed from the result, every failure and the server's standard error:
    as written, inside every JSON string literal however it is escaped, and in text escaped as JSON with
    no quotes around it, such as a log line, to four levels of string inside string. Which blocks are text
    is read before anything is removed. A failed http fetch is cleared of its proxy's credentials.

Tests that replace a file

Five tests that replace a file with the same bytes now write the new file before they remove the old
one. On a newer Linux runner, a removed file's inode went to the next file made, so nothing was
replaced and the identity checks had nothing to find.

Follow-ups

  • The OAP and Horizon read asz.view as the format pages say: the rule for a tool call that shares a
    record with text changed in docs/en/formats/asz-view.md.
  • Judging a goal as a whole is not in this change.

…tch references through MCP

A claim that something did not happen, a verdict of not observed, a question
never asked, an assumption never checked, or a person's goal found absent,
stays the evaluator's judgment. Beside it asz now reports how much of its
scope the evaluator was given, in steps: opened, only listed, not given, and
those whose evidence no one can be given. The counts come from what each
answer returned, counted once it was given; a navigation log that is not
whole makes them unavailable, never smaller.

The two divergence causes are renamed question_not_asked and
assumption_not_checked. A goal's first finding is its first recorded one,
chosen by one function for the close and -show. A person's goal is absent
only when every target was placed. A tool call that shares a record with the
agent's text reads as its arguments in asz.view.

A judge reference can be fetched by one tool of an MCP server, remote over
Streamable HTTP or local over standard input and output, speaking revision
2026-07-28 and the four revisions before it. asz judge -reference-tools lists
a server's tools. A URL is kept without its query, a server without its path,
and a failed fetch as a class and a detail asz builds. The secrets of a fetch
are removed from all it keeps, as written and through every JSON spelling.

Tests that replace a file write the new file before they remove the old one.
On a newer Linux runner, a removed file's inode went to the next file made,
so nothing was replaced and the identity checks had nothing to find.
@wu-sheng wu-sheng added this to the 0.7.0 milestone Oct 28, 2026
@wu-sheng
wu-sheng merged commit 341f8c0 into main Oct 28, 2026
25 checks passed
@wu-sheng
wu-sheng deleted the judge-reading-mcp branch October 28, 2026 01:09
wu-sheng added a commit to apache/skywalking that referenced this pull request Oct 28, 2026
A step on one part of a record reads that part alone, as the AI
Sessionizer does since apache/skywalking-ai-sessionizer#94. The document
read a tool call that shares a record with the agent's text as that
text, and a queued command kept as data beside other text as that text
too. Now a tool call reads as its arguments, and a queued command as the
prompt inside its own data.

On a local Sessionizer root, 10 of 25 LangChain tool calls shared a
record with text, and 3 of 109,397 Claude Code ones. No test fixture
holds such a record, so the documents the tests compare with the
Sessionizer's are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant