feat(evaluations): preserve the tool trajectory for judges

[donei003]

Stacked on #63 — base is feature/ld-judges-phase3, so the diff here is just this change.

Problem

#63 made the run's handler the judge's invoker and gave the judge the row's input, output, and expected output. What it could not give the judge was how the agent got there. Handler packages record tool traffic onto OpenTelemetry spans and return only {output, usage}, so by the time a criterion ran, the calls the row made on its way to that output were gone.

"Did the agent call the right tool, in the right order, with the right arguments?" was therefore an unaskable question of an SDK-run evaluation — one that had just finished running the agent that answered it.

Approach

The runner records the trajectory itself. A new evaluations/trajectory.py wraps the caller's tool implementations once per row, before they are handed to the handler.

Wrapping there is what makes this work with every handler package without changing any of them: a handler still looks a tool up by the key the model named and calls it, exactly as before. The alternative — having each of the six handler packages return a trajectory — would have been six changes, six chances to drift, and nothing for a custom caller-supplied handler.

The recorded trajectory reaches judges two ways:

• {{message_history}} — the row input, then the trajectory, then the generated output, then the formatting instructions. The placement is load-bearing twice: it is where the calls happened, and message_history is the variable every judge cloned from the AI Library's default templates already references, so a trajectory rubric needs no new judge template.
• {{tool_trajectory}} — the same block on its own, for a rubric that asks about tool use without restating the conversation.

What a judge now sees:

Can I get a refund on order A1?

Tools available: lookup_order, issue_refund
Tool calls made while producing the response, in order:
1. lookup_order
   arguments: {"id":"A1"}
   result: order A1 shipped 2026-08-02 via UPS
2. issue_refund
   arguments: {"amount":19.99,"id":"A1"}
   error: refund window closed

That order shipped on Aug 2 and is outside the refund window.

Your response MUST be in valid JSON format with the following structure:
...

Properties pinned by tests

• The recorder observes; it never intervenes. A wrapped tool returns exactly what the original returned and raises exactly what the original raised. Calls past the 50-call recording cap still execute and are only counted — truncation drops the record, never the work, because a harness that changed the agent's behavior would no longer be evaluating the agent.
• A recorder belongs to one row. Rows generate concurrently against one shared tool map, so a shared recorder would splice one row's calls into another's and hand the judge a conversation that never happened. Tested with two rows whose calls interleave.
• Order is call-start order, not completion order. A judge asked whether the agent searched before it refunded is reading a sequence; appending on return would reorder it.
• A tool result stays literal in the judge prompt. A tool result is a new injection surface alongside the generated output, closed by feat(evaluations): add LD judge event support #63's existing rule: the judge config is passed unrendered and the handler makes exactly one parse_template pass, so a {{...}} sequence coming back from a tool is never expanded. Tested with a tool that returns the literal text {{expected_output}}.
• Back-compatible for existing judges. A run with no observable tools adds no block and tool_trajectory is "", so a judge authored before this reads exactly the message_history it read before.
• Nothing is added to any event payload. Extended the existing exclusion assertion on the generation event. The trajectory reaches LaunchDarkly only inside the prompt a judge was shown, never as a wire field the backend has not specified.

Design notes

• Native provider tools are passed through unwrapped and left out of the rendered "tools available" line. They execute inside the provider, so no local wrapper ever sees them. Naming a tool whose use cannot be shown would invite a judge to conclude the model ignored a tool it may well have called.
• A row that had tools and called none says so explicitly ("No tool calls were made…"). "Called nothing" is the finding a tool-selection rubric most needs, and an omitted block would read as a run that had no tools at all.
• Bounds: 50 recorded calls per row, 2000 characters per rendered argument bag or result. A trajectory is interpolated into a judge prompt, so an agent looping over a large result set would otherwise spend the judge's context window — and budget — on a tail no judge reads.
• The judge path still receives the unwrapped tool map, so a judge's own tool calls cannot pollute the row's trajectory.
• Argument bags are serialized with sorted keys for determinism, so the rendered order is not the caller's argument order.
• row_fields() / render_row_trajectory() keep the row-record key names in one module: a key renamed on the write side without its reader would silently render every trajectory as empty, which reads exactly like an agent that called no tools.

Deliberately out of scope

Scorers have no access to the trajectory. Scorer.fn(row, output) is the contract, and the trajectory is not dataset-owned so it does not belong on DatasetRow. A deterministic trajectory check ("called lookup_order exactly once") is a natural follow-up but needs a contract change, not a quiet signature widening.

Validation

• uv run pytest -q — 1194 passed, 11 skipped
• uv run mypy packages/client/src/launchdarkly_ai_server/evaluations — clean
• uv run ruff format --check / uv run ruff check — clean
• 13 new unit tests in test_evaluations_trajectory.py, 6 new end-to-end tests in test_evaluations_run.py

Language-agnostic spec: launchdarkly/ai-sdks-monorepo#13 (new §8.7.4, amended §8.8.3).

🤖 Generated with Claude Code