feat(evaluations): add LD judge event support by donei003 · Pull Request #63 · launchdarkly/python-ai-sdk

donei003 · 2026-09-02T19:36:58Z

Summary

add Judge and Scorer criterion references for SDK-run evaluations, passed via run(criteria=[...])
resolve LD Judge configs (with a valid {kind: "evaluation"} context) before mutating evaluation/run state; resolution failures fail fast and carry the underlying cause
write judge/scorer criteria onto evaluation creation
execute each criterion per generated row — concurrently, bounded by the same concurrency parameter as generation — and emit one $ld:ai:offline-evals:evaluation event per (row, criterion) result
include criterion identity, judge metadata, validated score/reason, usage, timings, and cause-coded error payloads in evaluation events

Design notes

Public parameter is criteria= (union type Criterion = Judge | Scorer) to match the wire format (criteria / criterionType); a scorer is not a judge.
Scorer.fn receives the public DatasetRow dataclass plus the generated output, so internal result-dict keys are not part of the customer contract.
Judge configs are passed to the handler unrendered; the handler owns the single parse_template pass, so {{...}} sequences inside generated output or dataset values are never expanded into judge prompts.
Judge scores are validated when the judge responds (numeric, finite, 0–1). Invalid output becomes a per-criterion ERROR event with a cause code (invalid_judge_output, invalid_score, handler_raised, scorer_raised, generation_incomplete) and a top-level errorMessage, mirroring generation events — it never aborts the run after LLM spend. Queued generation events are flushed in a finally.
Duplicate criterion identities (judge key / scorer name collisions) are rejected before any records are created, since criterionType is part of the deterministic event identity.
The judge response contract (formatting instructions + {score, reasoning} parsing) is shared with the online judge path via judge_scoring.py.
Event payloads are frozen dataclasses with explicit wire serialization — no new runtime dependencies (the earlier pydantic dependency was dropped).

Notes

No pre-built LD Judges are created or hardcoded by the SDK; callers pass judge keys directly.
Evaluation results are emitted as individual LD events per (row, criterionType), not via batch ingest.
Still draft pending backend contract review: event/criteria naming and the cause-code taxonomy for error.code (string cause codes here vs. numeric 5001 on generation events).

Validation

uv run ruff format . / uv run ruff check .
uv run pytest -q — 1151 passed, 11 skipped
uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/judge_scoring.py packages/client/src/launchdarkly_ai_server/judges.py packages/client/src/launchdarkly_ai_server/__init__.py
Verified against the real ldclient that the judge-resolution context is valid (Context.from_dict({"kind": "evaluation", "key": ...}).valid is True)

Full uv run mypy . currently fails before checking due to duplicate test conftest module names across packages, which appears unrelated to this change.

Note

Overview
Offline init_evaluations().run() is no longer generation-only: callers can pass criteria mixing Judge (existing LaunchDarkly AI Judge configs) and Scorer (local sync/async functions on DatasetRow + output). Criteria are written on evaluation creation, judges are resolved and matched to handlers before any management API records exist, and each generated row gets one $ld:ai:offline-evals:criterion event per criterion (plus the existing generation events). The SDK emits scores only—pass/fail at ingest stays on LaunchDarkly—and result.passed now treats failed_rows as a gate failure when criteria are used.

Handler routing adds optional judge_handlers when a judge’s provider/mode differs from generation, with the same provider-priority rules as online judges (exact provider over wildcard, agent fallback for messages-mode judges). Judge prompts use a single handler-owned template pass; shared judge_scoring parsing is reused from judges.py. Generation event emission moves from stdout to INFO logging on launchdarkly_ai_server.evaluations.runner, evaluation runs no longer send rowCount, and criterion event delivery failures still flush queued events then raise so runs don’t poll silently.

^{Reviewed by Cursor Bugbot for commit ceca049. Bugbot is set up for automated code reviews on this repo. Configure here.}

donei003 · 2026-09-02T20:39:06Z

Addressed the feedback in the draft:

Added pydantic event DTOs in evaluations/events.py:
- EvaluationEventPayload base DTO for shared fields (projectKey, evaluationRunId, datasetKey, etc.)
- LDJudgeEvaluationEventPayload for LD Judge events
- DeterministicScorerEvaluationEventPayload for local scorer events
  Event emission now serializes those DTOs via model_dump(by_alias=True, exclude_none=True).
Removed user-supplied Judge.version; event version now only comes from the resolved LD variation metadata.
Judge resolution now passes an empty context to extract_variation for this phase.

Validation after changes:

uv run ruff format .
uv run ruff check .
uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
uv run pytest -q — 1128 passed, 11 skipped

donei003 · 2026-09-02T21:14:49Z

Updated for the usage feedback:

Added a dedicated pydantic TokenUsage DTO with inputTokens / outputTokens aliases.
Moved usage off the shared base event DTO and onto LDJudgeEvaluationEventPayload only.
Deterministic scorer events now cannot carry usage through their DTO (extra="forbid") and the test asserts no usage field is emitted for scorers.

Validation:

uv run ruff format .
uv run ruff check .
uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
uv run pytest -q — 1128 passed, 11 skipped

donei003 · 2026-09-02T21:56:26Z

Updated status from a Literal to an EvaluationStatus enum on the base event DTO. The pydantic config uses enum values when serializing the track payload, so emitted events still send "COMPLETE" / "ERROR".

Validation:

uv run ruff format .
uv run ruff check .
uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
uv run pytest -q — 1128 passed, 11 skipped

donei003 · 2026-09-02T21:56:59Z

Updated kind to use an enum as well:

Added EvaluationEventKind enum with JUDGE / SCORER values.
Base event DTO now has kind: EvaluationEventKind.
Judge/scorer event DTOs set their default kind from the enum.
Serialization still emits "judge" / "scorer" via use_enum_values=True.

Validation:

uv run ruff format .
uv run ruff check .
uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
uv run pytest -q — 1128 passed, 11 skipped

donei003 · 2026-09-02T22:15:41Z

Updated based on the latest feedback:

Moved extract_variation to a top-level import in evaluations/runner.py.
Simplified _resolve_judges now that we pass an empty context; it no longer accepts unused project_key / run_key args.
Removed both rowCount and expectedCriteriaCount from the evaluation run creation payload. The run payload is now just source + datasetId.
Updated tests accordingly.

Validation:

uv run ruff format .
uv run ruff check .
uv run mypy packages/client/src/launchdarkly_ai_server/evaluations packages/client/src/launchdarkly_ai_server/__init__.py
uv run pytest -q — 1128 passed, 11 skipped

Rename the public run() parameter judges= to criteria= (with JudgeReference -> Criterion and evaluations/judges.py -> criteria.py): the wire format already calls these criteria, and a Scorer is not a judge. Scorer callbacks now receive the public DatasetRow instead of the internal result dict, so internal key renames cannot break customer scorers; Criterion and DatasetRow are exported. Extract the judge response contract (formatting instructions, JSON score parsing, finite-number guard) into judge_scoring.py shared by the online judge path and the offline evaluations runner. The two copies had already drifted textually. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dering - Resolve LD judges with a valid {kind: evaluation, key: project} context. The previous empty context is invalid to the real LD SDK, so every resolution returned the None default and failed as 'not found'. The resolution error now also carries the underlying cause instead of always claiming the judge does not exist. - Validate judge scores when the judge responds: non-JSON output, non-numeric, non-finite, and out-of-range scores become per-criterion ERROR events with cause codes (invalid_judge_output, invalid_score, handler_raised, generation_incomplete, scorer_raised) instead of crashing the run at event-build time after all LLM spend. - Pass judge configs to the handler unrendered. The handler owns the single template pass, so {{...}} sequences inside generated output or dataset values can no longer be expanded into the judge prompt. Absent judge variables render as empty strings rather than leaving literal mustache in the prompt. - Reject duplicate criterion identities (judge keys / scorer names) before any records are created; they would share event identity. - Flush queued generation events in a finally so they reach LaunchDarkly even when the criteria phase fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rewrite the evaluation event payloads as frozen dataclasses with an explicit to_track_payload(), matching the Usage/RunSummary wire pattern used everywhere else in the SDK, and remove the pydantic>=2 dependency. The models validated the SDK's own dicts, and a validation failure surfaced as a run-aborting crash at emission time; payload construction is now also wrapped per result so one bad criterion result is logged and skipped instead of dropping the whole batch. ERROR events carry a top-level errorMessage for parity with generation events. Run (row x criterion) pairs through the same ConcurrencyController and concurrency parameter the generation phase uses, instead of one criterion at a time: a 200-row dataset with 3 judges was 600 serial LLM calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Emit per-event telemetry lines through the module logger instead of printing to the host application's stdout, refresh the run() docstring (no longer generation-only), and cover the shared judge response parser with unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The SDK has no way to learn whether a judge is lower-is-better or upper-is-better today. Gonfalon is adding isInverted as a top-level key in the flag-variation payload (same delivery path already used for variationKey/version); this wires the client side up to consume it. - ResolvedJudge gains is_inverted, read from the resolved variation's raw config dict (already passed through untouched by extract_variation/parse_ai_config, so no change needed there). - _resolve_judges populates it via config.get("isInverted"), tolerant of the key being absent (older Gonfalon) or non-bool. - LDJudgeEvaluationEventPayload gains success_direction, emitted as successDirection only when non-None, matching how usage/version are conditionally included. - _emit_evaluation_events maps is_inverted -> "lower_is_better" / "upper_is_better" / omitted (unresolved), never trusting client-side direction for verdict computation - only for what gets reported. Direction never affects the score itself; it's forwarded for ClickHouse ingestion in ai-evaluator, which is deferred/out of scope here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Course change: rather than reporting a raw score/threshold/direction for ai-evaluator to compute a verdict from server-side, the SDK now computes pass/fail itself using the judge's resolved isInverted, and sends that verdict directly. A missing threshold still means "no verdict" (score-only reporting), and an unresolved direction defaults to upper-is-better, matching Gonfalon's own default for a judge with no stored isInverted. verdict lives on the shared EvaluationEventPayload base class rather than just the judge payload, since deterministic scorers will need the same field once they compute their own verdicts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

## Summary Companion to launchdarkly/gonfalon#71281, which serves a judge's `isInverted` in the flag-evaluation payload the SDK already fetches. Course change from the initial approach: rather than reporting a raw score/threshold and a `successDirection` hint for ai-evaluator to compute a verdict server-side, the SDK now computes `verdict` (`pass`/`fail`) itself and sends that directly: - `evaluations/types.py`: `ResolvedJudge` gains `is_inverted: bool | None`. - `evaluations/runner.py::_resolve_judges`: captures `isInverted` off the resolved config (absent/non-bool resolves to `None`, not an error). - `evaluations/runner.py::_run_ld_judge_for_result`: when the `Judge` has a `threshold`, computes `verdict` by comparing `score` against `threshold` using `is_inverted` (unresolved direction defaults to upper-is-better, matching Gonfalon's own default for a judge with no stored `isInverted`). No threshold set means no verdict — score-only reporting. - `evaluations/events.py`: `verdict` lives on the shared `EvaluationEventPayload` base class (not just the judge payload), since deterministic scorers will need the same field once they compute their own verdicts. Emitted as `"verdict"` on the wire, omitted when `None`. This also resolves the "how would deepeval avoid a hardcoded metric→direction table" question from review: deepeval's `metric.success` is already a computed boolean from the library itself — if/when deepeval execution moves into the SDK, it reports that boolean directly, the same pattern this PR establishes for judges. ## Test plan - [x] `uv run pytest -k judge` — 51 passed, including 5 new parametrized verdict cases (upper/lower-is-better × pass/fail, plus unresolved-direction default) - [x] Full suite: 1156 passed, 11 skipped, no regressions - [x] `mypy`, `ruff check`, `ruff format --check` clean via LD Research 🤖

Reverses the client-side verdict from #78 and finishes the wire contract around it. Why the course change back: verdict policy will keep moving -- run-level pass rates are already specified and unbuilt, and warn bands and per-criterion policy are the obvious next asks. Anything the SDK computes is frozen at each customer's installed version and cannot be re-derived, because only the answer was stored. Server-side it is one implementation that applies to every SDK version and to runs already recorded. The reason #78 gave for moving it into the SDK -- that ai-evaluator has no way to learn a judge's direction -- is answered by hydrating successDirection onto the criterion through the Gonfalon proxy instead, the same way tool versions are already pinned there. - runner/events/types: judge results carry score and reason, never a verdict; ResolvedJudge drops is_inverted and _resolve_judges no longer reads isInverted off the served config. - Event key is now $ld:ai:offline-evals:criterion, matching the worker that polls for it; payload classes renamed to match. The previous ":evaluation" name was never read by anything, so nothing was ingested. - criteria: the create body carries kind and judgeKey, without which ai-evaluator defaults the criterion to deepeval and either rejects a judge key outright or silently registers "toxicity" as a server-judged metric and dispatches it to the judging worker. - Scorer declares success_direction (default higher_is_better). A scorer has no AI Config for the proxy to read a direction from, so the caller is the only source, and ai-evaluator needs one to rule at all. - Judge.threshold defaults to 0.5. A criterion with no threshold leaves nothing to compare a score against, so it would be stored and never ruled on. - The gate now counts failed_rows. It was omissible while runs were generation-only -- a row either generated or errored, and nothing produced a fail -- and with criteria it is the normal way a run fails, so a run where every row failed its judge was exiting 0. This reverses test_generation_failed_rows_do_not_fail_the_result, which asserted the old behavior without stating a rationale. - New test pins that judges resolve once per run, not per row: extract_variation reads flag delivery, so per-row resolution would let a mid-run edit change rubric, model, and provider between rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

## Summary Reverses the client-side verdict from #78 and finishes the wire contract around it. Targets `feature/ld-judges-phase3` (so it stacks under #63). **Why the course change back.** Verdict policy will keep moving — run-level per-judge pass rates are already specified in the Phase 3 tech spec and unbuilt, and warn bands and per-criterion policy are the obvious next asks. Anything the SDK computes is frozen at each customer's installed SDK version and **cannot be re-derived**, because only the answer was stored. Server-side it's one implementation that applies to every SDK version *and* to runs already recorded. The reason #78 gave for moving the comparison into the SDK — that ai-evaluator has no way to learn a judge's success direction — is answered instead by hydrating `successDirection` onto the criterion through the **Gonfalon proxy**, the same mechanism that already version-pins tool refs (`NewToolsValidationMiddleware`). That also keeps the one input a verdict is ruled on server-attested, even though the score beside it is client-reported. Companions: launchdarkly/ai-evaluator#693 (criterion carries `successDirection`), launchdarkly/ai-evaluator#699 (a non-error result must carry a verdict), a Gonfalon PR for the proxy middleware. launchdarkly/gonfalon#71281 is no longer needed and can close — the SDK doesn't read `isInverted` any more. ## Contract fixes this also carries Three of these were silent breakages between the SDK and the backend, not refinements: | Fix | Without it | |---|---| | Event key → `$ld:ai:offline-evals:criterion` | The worker in ai-evaluator#692 polls for `:criterion`; the SDK emitted `:evaluation`. **Nothing was ever ingested.** | | Criteria body carries `kind` + `judgeKey` | ai-evaluator defaults an absent `kind` to `deepeval`, whose enum constrains `criterionType` — so a judge key is **rejected at create**, and `criterionType="toxicity"` is silently registered as a server-judged metric and dispatched to the judging worker. | | Gate counts `failed_rows` | `passed` was `error_rows == 0 and pending_rows == 0`. A run where **every row failed its judge exited 0.** | ## Also - **`Scorer.success_direction`** (default `higher_is_better`). A scorer has no AI Config for the proxy to read a direction off, so the caller is the only source — and ai-evaluator needs one to rule at all. Set `lower_is_better` for a scorer counting something unwanted (regex hits, edit distance). - **`Judge.threshold` defaults to 0.5.** A criterion with no threshold leaves nothing to compare a score against, so it would be stored and never ruled on. (#78's "no threshold means no verdict" is not survivable now that a NULL verdict is how ai-evaluator marks an error.) - **New test: judges resolve once per run, not per row.** `extract_variation` reads flag delivery, an in-memory store that updates within seconds of a UI edit, so per-row resolution would let a mid-run edit change rubric text, judge model, and provider *between rows of one run*. The online path (`judges.build_judge_tasks`) does resolve per invocation, so reusing it for convenience is a live way to reintroduce this. ## ⚠️ One deliberate behavior reversal `test_generation_failed_rows_do_not_fail_the_result` asserted that failed rows keep `passed is True`. It's now `test_failed_rows_fail_the_result` and asserts the opposite. That assertion arrived with an unrelated polling fix and stated no rationale; it was unobservable when runs were generation-only (a row either generated or errored, and nothing produced a "failed"). Worth a second opinion if it was load-bearing for a reason not written down. ## Test plan - [x] `uv run pytest packages/client/tests -q` — 402 passed - [x] Replaced the verdict-computation test with its inverse: parametrized over `isInverted` (including the served-payload value) to pin that the SDK does not compare **even when it could** - [x] New: scorer `lower_is_better` reaches the wire verbatim; judge threshold defaults onto the criteria body; judges resolve once across a 3-row run - [x] `mypy` clean over `evaluations/`, `judge_scoring.py`, `judges.py`, `__init__.py` - [x] `ruff check` / `ruff format` clean via LD Research 🤖

The API's own duplicate-criterion check lowercases criterionType before comparing (the worker's retry gate does the same), so two criteria differing only by case still collide server-side. The SDK's local fail-fast check compared case-sensitively, so a case-variant duplicate would pass the SDK check, pay for judge resolution and a dataset fetch, and only then be rejected by the API. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Dylan has a gonfalon-judge-proxy PR adding the successDirection injection middleware but its enabling flag is off, so ai-evaluator's create-time validation 400s on a judge criterion with no successDirection. Hardcoding higher_is_better on the wire unblocks testing that proxy PR end to end. DO NOT MERGE THIS COMMIT: Judge.to_criteria_wire deliberately omits successDirection so the proxy can inject a server-attested value from the judge's AI Config; revert this alongside its two updated test assertions once the proxy lands. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…_INSTRUCTIONS _judge_variables built message_history from just input+output, then exposed FORMATTING_INSTRUCTIONS as a separate formatting_instructions variable. But no judge -- including every one of the AI Library's default templates (accuracy, relevance, toxicity) and anything cloned from them -- references {{formatting_instructions}}; they all reference {{message_history}}, because judges.run_judges (the online path) already bakes FORMATTING_INSTRUCTIONS into that value: message_history = "\n\n".join(filter(None, [user_input, llm_response, FORMATTING_INSTRUCTIONS])) Offline diverged from that and never asked the model for the {score, reasoning} JSON shape, so every judge response came back as free-text prose and failed to parse (invalid_judge_output) -- for any judge built the standard way, not just a misconfigured one. Match the online construction so an unmodified default-template judge scores correctly through both paths, per judge_scoring.py's own "the two paths cannot drift" contract. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 3 potential issues.

3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-09-14T21:08:04Z

+            result = await handler(
+                dict(resolved.config),
+                row.get("output"),
+                tool_handlers,
+                {
+                    **variables,
+                    "formatting_instructions": FORMATTING_INSTRUCTIONS,
+                },
+            )


🟡 Judge mode mismatch reaches handler

For a same-provider judge with another mode, _select_judge_handler still selects the generation handler. The fallback checks only provider, so incompatible judge configs reach that handler.

Learn more

Handlers declare both a provider and an execution mode. Normal routing requires both values to match, except the supported messages-to-agent fallback. This branch checks only the provider before selecting the generation handler. An agent judge can therefore run through a messages handler, although no reverse conversion makes that config compatible.

Example: The generation handler provides ("OpenAI", "messages"), while an OpenAI judge resolves in "agent" mode. The branch selects the messages handler instead of rejecting the run for missing agent coverage.

Recommended fix: Require generation_provides_for[1] == resolved.mode, or the explicit agent-handler fallback for a messages judge, before returning JudgeExecution. Otherwise return None so _resolve_judges reports the missing compatible handler.

Was this helpful? React with 👍 or 👎 to provide feedback.

Address PR #63 review feedback. A judge is an independent AI Config and may be served by a different provider or mode than the evaluation's generation config, but the offline path invoked the generation handler for every judge. An OpenAI handler handed an Anthropic judge config called OpenAI with an Anthropic model name, so affected rows recorded handler_raised instead of a score -- after the generation spend had already happened. ResolvedJudge now carries the judge's own provider and normalized mode, and each judge is paired with a handler at resolution time, before any records are created: a judge handler matching its provider and mode, an agent-mode handler for a messages-mode judge (with messages collapsed, as the online path does), or the generation handler when it covers the judge's provider. A judge no handler covers fails the run up front. Handlers for other providers are passed as run(judge_handlers=[...]); a plain callable with no provides_for still routes itself, unchanged. Also from review: - Criterion event emission failures are collected and raised together after every result is attempted. Dropping them left the backend short of a result per (row, criterion), so run() polled to its 180s timeout and hid the tracking exception. - NaN thresholds are rejected. Both range comparisons are false for NaN, so it reached the wire as a bare NaN literal the management API rejects. - run_judge no longer coerces a null judge score to 0.0. It recorded a gen_ai.evaluation of 0 -- indistinguishable from a hard fail -- where every other non-numeric judge output skips the metric. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…testing" This reverts commit 16b5d22. The hardcode existed only to get past ai-evaluator's create-time validation while the gonfalon-judge-proxy injection middleware was behind an off flag. Judge.to_criteria_wire omits successDirection again so the proxy hydrates it from the judge's AI Config: the one input a verdict is ruled on stays server-attested, rather than being asserted by whichever SDK version happened to create the evaluation. A Scorer still declares its own direction, since it has no LaunchDarkly-side config to read one from. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Address PR #63 review feedback. _select_judge_handler took the first judge handler whose provider matched or was a wildcard, so with judge_handlers=[LangChain("*"), OpenAI("OpenAI")] an OpenAI judge ran through the LangChain adapter and reversing the list changed which handler scored it. A wildcard is a fallback for multi-provider adapters: the handler contract gives an exact provider name priority, and config() already routes generation configs that way. Handler lookup is now two passes -- exact provider, then wildcard -- applied to both the judge's own mode and the messages-to-agent fallback, so the order a caller lists its handlers in no longer decides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The evaluations section still described a generation-only harness, so the criteria this PR adds were undocumented and two claims had gone stale: events are logged on launchdarkly_ai_server.evaluations.runner rather than printed to stdout, and EvalRunResult.passed now treats failed_rows as a gate failure. Adds a "Score rows with judges and scorers" section covering the Judge and Scorer contracts and their threshold defaults, the criterion event, why the SDK reports a score and LaunchDarkly rules on it, per-judge handler routing via judge_handlers, and which failures fail the run up front versus becoming a cause-coded per-criterion ERROR event. The barrel package README picks up the criterion types it re-exports and the second event name. Both snippets were executed against the package: the imports resolve, and the judge criterion wire payload carries no successDirection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The examples passed "$ld:ai:judge:accuracy", a key from LaunchDarkly's own internal namespace, which reads as though callers must reach for that prefix. A judge key is just the key of a judge in the reader's project, so the examples now use "accuracy-judge". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

donei003 force-pushed the feature/ld-judges-phase3 branch from e7fffd7 to 614adca Compare September 2, 2026 20:38

donei003 force-pushed the feature/ld-judges-phase3 branch from 614adca to 6637d48 Compare September 2, 2026 21:14

donei003 force-pushed the feature/ld-judges-phase3 branch from 6637d48 to 6104ae0 Compare September 2, 2026 21:56

donei003 force-pushed the feature/ld-judges-phase3 branch from 6104ae0 to 7ed3d05 Compare September 2, 2026 21:56

feat(evaluations): add LD judge event support

ccfdc0a

donei003 force-pushed the feature/ld-judges-phase3 branch from 7ed3d05 to ccfdc0a Compare September 2, 2026 22:15

donei003 and others added 8 commits September 2, 2026 16:45

donei003 mentioned this pull request Sep 10, 2026

Report criterion scores and let LaunchDarkly rule on them #79

Merged

5 tasks

donei003 and others added 4 commits September 9, 2026 21:37

donei003 marked this pull request as ready for review September 14, 2026 21:05

donei003 requested review from a team, aknight-ld and andrewklatzke September 14, 2026 21:05

devin-ai-integration Bot reviewed Sep 14, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

donei003 and others added 4 commits September 15, 2026 13:30

donei003 mentioned this pull request Sep 15, 2026

feat(evaluations): preserve the tool trajectory for judges #89

Draft

Navigation Menu

Uh oh!

feat(evaluations): add LD judge event support - #63

feat(evaluations): add LD judge event support#63
donei003 wants to merge 18 commits into
mainfrom
feature/ld-judges-phase3

donei003 commented Sep 2, 2026 •

edited by cursor Bot

Loading

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

devin-ai-integration Bot left a comment •

edited

Loading

Uh oh!

devin-ai-integration Bot Sep 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

donei003 commented Sep 2, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design notes

Notes

Validation

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

donei003 commented Sep 2, 2026

Uh oh!

devin-ai-integration Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

donei003 commented Sep 2, 2026 •

edited by cursor Bot

Loading

devin-ai-integration Bot left a comment •

edited

Loading

devin-ai-integration Bot Sep 14, 2026 •

edited

Loading