Repository navigation
Report a judge run by goal, and compare it with what a person concluded - #88
Merged
Merged
Conversation
Member
Author
Example: what
|
wu-sheng
force-pushed
the
judge-report
branch
4 times, most recently
from
October 23, 2026 08:41
da44665 to
d619669
Compare
A judge run now says what the conversation was for, whether each target was reached, why asz believes it, and what the agent did for it. - The targets stage says how each input relates to the inputs before it, and groups targets into goals, each with how its request changed and a main target, or undetermined. - The achievement stage records each target's deliveries, undoings and deliveries again, the contradictions tied to them, its rounds of rework, findings anchored where a problem happened, and an account of what was done: what was checked for a question, the file operations for a change, and the requests to another system. Excerpts are checked against the record. - Each target ends in a class, with the record's score and a final score that applies what contradicts or undoes the deliveries standing at the end. Rework is priced by how its calls were labeled. - File operations come from change records, and from the requests of editing tools, which each runtime's adapter reads. - asz judge -reference saves each source before any evaluator runs. A reference stage finds the conclusions it holds, and a divergence stage compares the conversation with each; a contradiction or a rollback of a standing delivery changes the final score. - asz judge -labels and -regroup compare the judge's grouping with a person's goals, a plain list in their own words: one evaluator call places each target in the goal it serves, and asz classes each goal, in a versioned file beside the run. - asz judge -show prints all of it goal by goal, the most severe first. - CI runs on ubuntu-26.04, and ubuntu-26.04-arm, rather than ubuntu-latest, so a new image never changes what CI runs on without a change here.
wu-sheng
force-pushed
the
judge-report
branch
from
October 23, 2026 09:33
d619669 to
cd862fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changes what a judge run reports. A person reading it should learn what the conversation was for,
whether each target was reached, why asz believes it, and what the agent actually did, not only counts
of calls. A run can also be compared with what a person concluded elsewhere, such as a ticket or a
review, and with a person's own grouping of the conversation.
What it adds
it starts a goal, goes on with one, adjusts one, returns to one, is unclear, or is not from the person.
Targets are grouped into goals from those relations, since a person often works on unrelated things
in one session. A goal keeps how its request changed, and its main target when the person's words
show one, or says it is undetermined.
delivered,undoneanddelivered_again, each shown by the reply or tool step that did it. Adelivery again names the deliveries it corrects, and an undoing the criteria and deliveries it took
back, so a result delivered in parts is followed part by part. Contradictions are tied to a delivery.
Each target ends in a class, most severe first: contradicted, reverted, not achieved, partly,
achieved, cannot be verified, or cancelled or replaced.
sj/1score as before. The final score applieswhat contradicts or undoes the deliveries that stand at the end. The report shows both when they
differ.
and the answer. For code: the file operations, from workspace change records, or from the requests
of editing tools, which each adapter decodes; the evaluator groups and summarizes them. For another
system: each operation with its request, its response and what confirmed it.
the record. A target under 100 lists what it still misses.
its cause, and what it cost, priced by how its calls were labeled. Findings are anchored at the step
where a problem happened, with a proposed improvement. asz adds its own for what the stages decided.
asz judge -reference SRCtakes a file, an http or https URL, or a source aconfigured command fetches, such as a ticket behind a login. Each source is saved in the run's
work/sources/before any evaluator runs, and the command never sees the conversation. Areferencestage finds the conclusions a source holds and how far each can be trusted, including aconclusion the source corrected later. A
divergencestage says how the conversation compares witheach: reached, contradicted, not observed, unknown or not applicable, with rollbacks the source
reports and why the work went another way. A contradiction or rollback of a standing delivery changes
the final score; one that does not is kept, and the report says why.
took, so
asz judge -labels FILEtakes a plain list of their goals in their own words, and-regroup Napplies it to a closed run. One evaluator call places each target in the goal itserves, citing the person's input that shows it, without seeing the judge's goals. asz then classes
each goal:
agree,split,merged,missed(worked on, no target found),absentorunknown.A regroup is a versioned file beside the run, which is never changed, and keeps the goal file's
text; scores are the same in both views.
asz judge -showprints all of it goal by goal, the most severe outcome first, with eachtarget's history. Text the evaluator wrote is marked
read:, apart from what asz checked.The
sj/1format is extended in place, since it is not released yet.CI also runs on
ubuntu-26.04, andubuntu-26.04-armfor the arm64 builds, rather thanubuntu-latest,so a new image never changes what CI runs on without a change here.
Docs
docs/en/concepts-and-designs/conversation-judge.md: goals, deliveries, outcome classes, scores,accounts, rework, findings, references and labels, with the runs behind each rule.
docs/en/formats/session-judgment.md: the new frames, the two new stages, and the regroup file.docs/en/setup/judge.mdandconfiguration.md:-reference,-labels,-regroupandjudge.references.docs/en/concepts-and-designs/glossary.md: goal and input relation.Tests
make checkpasses. Every new rule was made false in turn, and each change compiled before the testsran; a test fails for every one of them. Conditions found implied by another were removed.
The stages were also run with a real evaluator on a local conversation of three targets, with and
without a reference, and those runs changed the rules where the checks were stricter or looser than
the record:
that reports both.
aligned output on one line; an observation given twice is kept once, since one evaluator wrote the
same observation 97 times.
which deliveries stand: an evaluator called a closing summary a delivery again, and a source's
contradiction of the first answer then changed no score.
could reach its score.
A regroup of the same run by four goals in plain words, one never worked on, placed each target on the
input that asked for it, and two calls gave the same comparison.
An example of what
-showprints is in a comment below.