feat(evaluations): add LD judge event support

packages/ai/README.md

@@ -54,10 +54,10 @@ Never raises. Returns `{"enabled": bool, "config": dict | None, "meta": dict | N
 
 ## Evaluations from code
 
-`init_evaluations` and the evaluations result types are also re-exported:
+`init_evaluations`, the criterion types, and the evaluations result types are all re-exported:
 
 ```python
-from launchdarkly_ai_python import init_evaluations
+from launchdarkly_ai_python import Judge, Scorer, init_evaluations
 
 evals = init_evaluations()
 result = await evals.run(
@@ -66,10 +66,14 @@ result = await evals.run(
     dataset="golden-dataset",
     handler=my_handler,
     generation={"provider": "OpenAI", "model": "gpt-4o"},
+    criteria=[
+        Judge(key="accuracy-judge"),
+        Scorer(name="mentions-policy", fn=lambda row, output: "policy" in (output or "")),
+    ],
 )
 ```
 
-`LD_API_TOKEN` is required. Configure `LD_SDK_KEY` — or initialize your own client with `init_client(client=...)` — to emit one `$ld:ai:offline-evals:generation` event per generated row through the standard SDK event transport. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. Evaluation-run links use the explicit `ui_base_uri` option or `LD_UI_BASE_URI` (for example, `https://ld-stg.launchdarkly.com` in staging), defaulting to `https://app.launchdarkly.com`. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
+`LD_API_TOKEN` is required. Configure `LD_SDK_KEY` — or initialize your own client with `init_client(client=...)` — to emit one `$ld:ai:offline-evals:generation` event per generated row, plus one `$ld:ai:offline-evals:criterion` event per `(row, criterion)` when `criteria` are supplied, through the standard SDK event transport. The SDK reports scores; LaunchDarkly rules on them at ingest. A judge served by a different provider than `generation` needs a handler for it in `judge_handlers`. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. Evaluation-run links use the explicit `ui_base_uri` option or `LD_UI_BASE_URI` (for example, `https://ld-stg.launchdarkly.com` in staging), defaulting to `https://app.launchdarkly.com`. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
 
 ---
 

packages/client/README.md

@@ -46,9 +46,9 @@ No code changes are required — `init_client()` detects the packages at runtime
 
 ### Run an evaluation from code
 
-The generation-only evaluations harness reads an LD-hosted dataset, creates a new evaluation and API-source run, and invokes your handler once per row. Each success or error queues a `$ld:ai:offline-evals:generation` custom event containing the evaluation, run, dataset, and row identifiers plus output or error (`errorMessage` is included for `ERROR` rows), nested `usage.inputTokens`/`usage.outputTokens`, timing, and stable hashes. Dataset-owned input, expected output, metadata, and variables are not duplicated in the event. Each queued event prints a line to stdout with its RFC3339 UTC `emittedAt` timestamp and stable `eventId`, making it possible to compare SDK emission time with ClickHouse arrival time. The same `emittedAt` value is included in the event payload. Events are flushed before the summary is fetched and the call returns; handlers are never rerun to retry event delivery. Pass/fail is derived from LaunchDarkly's run summary.
+The evaluations harness reads an LD-hosted dataset, creates a new evaluation and API-source run, and invokes your handler once per row. Rows can then be scored by LaunchDarkly judges and local scorer functions; see [Score rows with judges and scorers](#score-rows-with-judges-and-scorers). Each success or error queues a `$ld:ai:offline-evals:generation` custom event containing the evaluation, run, dataset, and row identifiers plus output or error (`errorMessage` is included for `ERROR` rows), nested `usage.inputTokens`/`usage.outputTokens`, timing, and stable hashes. Dataset-owned input, expected output, metadata, and variables are not duplicated in the event. Each queued event is logged at `INFO` on the `launchdarkly_ai_server.evaluations.runner` logger with its RFC3339 UTC `emittedAt` timestamp and stable `eventId`, making it possible to compare SDK emission time with ClickHouse arrival time once that logger is enabled. The same `emittedAt` value is included in the event payload. Events are flushed before the summary is fetched and the call returns; handlers are never rerun to retry event delivery. Pass/fail is derived from LaunchDarkly's run summary.
 
-Result links use `ui_base_uri`, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; this is independent of `LD_API_BASE_URI`. After flushing generation events, the harness polls the run summary endpoint until passed + failed + error rows fully account for a nonzero total with no pending rows, polling every `poll_interval_seconds` (default 2s) up to `poll_timeout_seconds` (default 180s); pass either to `run()` to widen both for large datasets. The summary endpoint does not return run state, so `RunSummary` exposes row counts only. A generation result passes only when the completed summary has no error or pending rows. Evaluation keys must be unique because every call creates a new evaluation with `POST`.
+Result links use `ui_base_uri`, then `LD_UI_BASE_URI`, then `https://app.launchdarkly.com`; this is independent of `LD_API_BASE_URI`. After flushing generation events, the harness polls the run summary endpoint until passed + failed + error rows fully account for a nonzero total with no pending rows, polling every `poll_interval_seconds` (default 2s) up to `poll_timeout_seconds` (default 180s); pass either to `run()` to widen both for large datasets. The summary endpoint does not return run state, so `RunSummary` exposes row counts only. A run passes only when the completed summary has no failed, error, or pending rows. `failed_rows` counts rows whose criteria were scored and did not meet their threshold, so a gate that ignored it would exit 0 on a run where every row failed its judge. Evaluation keys must be unique because every call creates a new evaluation with `POST`.
 
 ```python
 import asyncio
@@ -80,7 +80,45 @@ sys.exit(asyncio.run(main()))
 
 `project_key` is supplied per run rather than during initialization. `generation.instructions` is shorthand for one system message; use `generation.messages` instead for a full message list, but do not supply both. The harness never retries a handler invocation because doing so could repeat tool side effects. Its retries apply only to LaunchDarkly management API requests.
 
-Generation events are the only path by which row results reach LaunchDarkly, so `init_evaluations()` raises rather than creating a run that can never complete unless it can resolve an event transport: either an SDK key (`sdk_key` or `LD_SDK_KEY`) or a client already initialized through `init_client(client=...)`. Bringing your own client lets a process emit evaluation events without an SDK key in scope. Every generated row is emitted and flushed unconditionally; no feature flag gates event publishing. The harness then polls the summary endpoint until row accounting shows processing is complete.
+Generation and criterion events are the only path by which row results reach LaunchDarkly, so `init_evaluations()` raises rather than creating a run that can never complete unless it can resolve an event transport: either an SDK key (`sdk_key` or `LD_SDK_KEY`) or a client already initialized through `init_client(client=...)`. Bringing your own client lets a process emit evaluation events without an SDK key in scope. Every generated row is emitted and flushed unconditionally; no feature flag gates event publishing. The harness then polls the summary endpoint until row accounting shows processing is complete.
+
+### Score rows with judges and scorers
+
+Pass `criteria` to `run()` to score every generated row. A `Judge` references an AI Judge config that already exists in LaunchDarkly — the SDK creates no judges and ships none of its own — and a `Scorer` wraps a local function, so a run can mix model-graded and deterministic checks. Each criterion runs once per generated row, bounded by the same `concurrency` as generation, and emits one `$ld:ai:offline-evals:criterion` event per `(row, criterion)` carrying the criterion identity, the judge's variation key and version, the validated score, its reason, usage, and timings.
+
+```python
+from launchdarkly_ai_claude_messages import create_claude_messages_handler
+from launchdarkly_ai_openai_messages import create_openai_messages_handler
+from launchdarkly_ai_server import DatasetRow, Judge, Scorer, init_evaluations
+
+
+def mentions_policy(row: DatasetRow, output: str | None) -> bool:
+    return "refund policy" in (output or "").lower()
+
+
+result = await init_evaluations().run(
+    project_key="my-project",
+    key="support-qa-2026-08-20",
+    dataset="support-golden",
+    handler=create_openai_messages_handler(),
+    generation={"provider": "OpenAI", "model": "gpt-4o"},
+    criteria=[
+        Judge(key="accuracy-judge", threshold=0.8),
+        Scorer(name="mentions-policy", fn=mentions_policy),
+    ],
+    # Needed only because this judge is served by a different provider than
+    # the generation config above.
+    judge_handlers=[create_claude_messages_handler()],
+)
+```
+
+`Scorer.fn` receives the `DatasetRow` the output was generated from plus the generated output, may be sync or async, and must return a bool or a number from 0 to 1; booleans become 1.0 or 0.0. `Judge.threshold` defaults to 0.5 and `Scorer.threshold` to 1.0 — a perfect score, which is what a boolean scorer wants — and both accept an optional `pass_rate_threshold`. Judge keys and scorer names share one `criterionType` namespace and must be unique within a run, case-insensitively, because that name is part of each result's deterministic event identity. `Judge.ground_truth_context` overrides what the judge is graded against when the dataset row's expected output is not it.
+
+**The SDK reports scores and never rules on them.** LaunchDarkly derives each row's verdict at ingest by comparing the score against the criterion's stored threshold and success direction, so pass/fail policy is one server-side implementation that applies to every SDK version and to runs already recorded. A judge's direction lives on its AI Config and is injected server-side, keeping the one input a verdict turns on server-attested; a `Scorer` has no LaunchDarkly-side config to read, so it declares its own `success_direction` (default `"higher_is_better"` — set `"lower_is_better"` for a scorer that counts something unwanted, like a regex hit count).
+
+**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`, and a handler built for one provider cannot execute another's config. `handler` runs a judge when it provides for that judge's provider; pass handlers for any other providers in `judge_handlers`. Selection prefers a handler naming the judge's provider outright over a wildcard multi-provider adapter, and an agent-mode handler can serve a messages-mode judge with its messages collapsed into one instructions block. A plain callable that declares no `provides_for` routes itself, exactly as it already does for the generation config.
+
+Judges are resolved through flag delivery, and handlers are matched to them, **before** any evaluation records are created — a missing judge or one no handler covers fails the run up front rather than after the generation spend. After that point a criterion failure never aborts the run: an unparseable judge response, an out-of-range score, a raising handler or scorer, and a row whose generation errored each become a per-criterion `ERROR` event with a cause code (`invalid_judge_output`, `invalid_score`, `handler_raised`, `scorer_raised`, `generation_incomplete`) and a top-level `errorMessage`. Event *delivery* is different: the backend needs one result per `(row, criterion)` to finish row accounting, so if tracking a criterion event fails, every remaining result is still attempted and flushed and then `run()` raises — rather than polling to its timeout with the cause hidden.
 
 The client uses **lazy initialization**: importing the package does not connect to LaunchDarkly. The singleton is created automatically on the first API call that needs it (`config().invoke()`, `graph().invoke()`, `resolve_graph()`, etc.), as long as `LD_SDK_KEY` is set in the environment.
 

packages/client/src/launchdarkly_ai_server/__init__.py

@@ -22,11 +22,15 @@
     set_conversation_id_if_absent,
 )
 from .evaluations import (
+    Criterion,
+    DatasetRow,
     EvalRunResult,
     EvaluationsError,
     EvaluationsModule,
     GenerationConfig,
+    Judge,
     RunSummary,
+    Scorer,
     init_evaluations,
 )
 from .graph import GraphInstance, graph, resolve_graph
@@ -165,10 +169,14 @@
     "VariationMeta",
     # evaluations
     "EvalRunResult",
+    "Criterion",
+    "DatasetRow",
     "EvaluationsError",
     "EvaluationsModule",
     "GenerationConfig",
+    "Judge",
     "RunSummary",
+    "Scorer",
     "init_evaluations",
     # utils
     "create_handler",

packages/client/src/launchdarkly_ai_server/evaluations/__init__.py

@@ -9,19 +9,25 @@
     Transport,
     urllib_transport,
 )
+from .criteria import Criterion, Judge, Scorer, SuccessDirection
 from .module import EvaluationsModule, init_evaluations
-from .types import EvalRunResult, GenerationConfig, RunSummary, Usage
+from .types import DatasetRow, EvalRunResult, GenerationConfig, RunSummary, Usage
 
 __all__ = [
     "DEFAULT_BASE_URI",
+    "Criterion",
+    "DatasetRow",
     "EvalRunResult",
     "EvaluationsError",
     "EvaluationsModule",
     "GenerationConfig",
     "HttpResponse",
+    "Judge",
     "LDApiClient",
     "LDApiError",
     "RunSummary",
+    "Scorer",
+    "SuccessDirection",
     "Transport",
     "Usage",
     "init_evaluations",

packages/client/src/launchdarkly_ai_server/evaluations/criteria.py

@@ -0,0 +1,155 @@
+from __future__ import annotations
+
+from collections.abc import Awaitable, Callable
+from dataclasses import dataclass
+from math import isnan
+from typing import Any, Literal
+
+from .types import DatasetRow
+
+type ScorerFn = Callable[[DatasetRow, Any], float | bool | Awaitable[float | bool]]
+
+type SuccessDirection = Literal["higher_is_better", "lower_is_better"]
+
+#: The threshold a judge is held to when its reference sets none. A criterion
+#: with no threshold gives LaunchDarkly nothing to compare a score against, so
+#: it would be recorded and never ruled on; defaulting is what keeps a judge
+#: usable as a gate without every caller restating the obvious.
+DEFAULT_JUDGE_THRESHOLD = 0.5
+
+
+@dataclass(frozen=True)
+class Judge:
+    """Reference to a LaunchDarkly AI Judge config to run for each eval row.
+
+    The SDK does not create or provide built-in judges. Pass the key of a judge
+    that exists in LaunchDarkly. Resolution uses LaunchDarkly flag delivery for
+    the currently served variation.
+
+    A judge's success direction is not set here. It lives on the judge's AI
+    Config as ``isInverted`` and is injected onto the criterion by LaunchDarkly
+    when the evaluation is created, so the one input a verdict is ruled on stays
+    server-attested even though the score beside it is client-reported.
+    """
+
+    key: str
+    threshold: float | None = DEFAULT_JUDGE_THRESHOLD
+    pass_rate_threshold: float | None = None
+    ground_truth_context: str | None = None
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.key, str) or not self.key.strip():
+            raise ValueError("judge key must not be blank")
+        _validate_thresholds(
+            threshold=self.threshold,
+            pass_rate_threshold=self.pass_rate_threshold,
+        )
+
+    @property
+    def criterion_type(self) -> str:
+        return self.key
+
+    def to_criteria_wire(self) -> dict[str, Any]:
+        options = _criteria_options(
+            threshold=self.threshold,
+            pass_rate_threshold=self.pass_rate_threshold,
+            ground_truth_context=self.ground_truth_context,
+        )
+        return {
+            "criterionType": self.criterion_type,
+            "kind": "judge",
+            "judgeKey": self.key,
+            "options": options,
+        }
+
+
+@dataclass(frozen=True)
+class Scorer:
+    """Local deterministic scorer run for each generated evaluation row.
+
+    ``fn`` may be sync or async and receives ``(row, output)``, where ``row``
+    is the :class:`~launchdarkly_ai_server.evaluations.types.DatasetRow` the
+    output was generated from and ``output`` is the generated output. It must
+    return a boolean or a numeric score from 0 to 1. Boolean results are
+    converted to 1.0 or 0.0 before being emitted as evaluation events.
+
+    ``threshold`` defaults to 1.0: a row passes only on a perfect score, which
+    matches the common case of boolean scorers. Pass a lower threshold for
+    graded numeric scorers.
+
+    ``success_direction`` says which way the score points, and defaults to
+    higher-is-better. Unlike a judge, a scorer has no LaunchDarkly-side config
+    to read a direction from, so the declaration here is the only source --
+    LaunchDarkly derives each row's verdict by comparing score to threshold in
+    this direction. Set ``"lower_is_better"`` for a scorer that counts
+    something unwanted, e.g. a regex hit count or an edit distance.
+    """
+
+    name: str
+    fn: ScorerFn
+    threshold: float | None = 1.0
+    pass_rate_threshold: float | None = None
+    success_direction: SuccessDirection = "higher_is_better"
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.name, str) or not self.name.strip():
+            raise ValueError("scorer name must not be blank")
+        if not callable(self.fn):
+            raise ValueError("scorer fn must be callable")
+        _validate_thresholds(
+            threshold=self.threshold,
+            pass_rate_threshold=self.pass_rate_threshold,
+        )
+
+    @property
+    def criterion_type(self) -> str:
+        return self.name
+
+    def to_criteria_wire(self) -> dict[str, Any]:
+        return {
+            "criterionType": self.criterion_type,
+            "kind": "scorer",
+            "successDirection": self.success_direction,
+            "options": _criteria_options(
+                threshold=self.threshold,
+                pass_rate_threshold=self.pass_rate_threshold,
+            ),
+        }
+
+
+type Criterion = Judge | Scorer
+
+
+def _validate_thresholds(
+    *,
+    threshold: float | None,
+    pass_rate_threshold: float | None,
+) -> None:
+    for name, value in (
+        ("threshold", threshold),
+        ("pass_rate_threshold", pass_rate_threshold),
+    ):
+        if value is None:
+            continue
+        # NaN passes both range comparisons, so without an explicit check it
+        # reaches to_criteria_wire and is serialized as a bare ``NaN`` literal
+        # the management API rejects -- an evaluation that fails to be created
+        # rather than a threshold that fails to validate.
+        if isnan(value) or value < 0 or value > 1:
+            raise ValueError(f"{name} must be a number between 0 and 1")
+
+
+def _criteria_options(
+    *,
+    threshold: float | None,
+    pass_rate_threshold: float | None,
+    ground_truth_context: str | None = None,
+) -> dict[str, Any]:
+    options: dict[str, Any] = {}
+    if threshold is not None:
+        options["threshold"] = threshold
+    if pass_rate_threshold is not None:
+        options["passRateThreshold"] = pass_rate_threshold
+    if ground_truth_context is not None:
+        options["groundTruthContext"] = ground_truth_context
+    return options