Add LLM-as-Judge observability blog

[peachisai]

Add English and Chinese guides explaining how LLM-as-Judge connects GenAI response quality with SkyWalking evaluation records, metrics, alarms, and source traces. Include configuration examples and screenshots.

The guides target the upcoming SkyWalking 11.1.0 release, explain the current development-build requirement, and use a publication date of 2026-10-15.

Validation: checked image references and guide links, rendered both articles in a minimal Hugo 0.159.0 fixture, and ran git diff --check. A full site build was not run locally.

[netlify]

✅ Deploy Preview for skywalking-website-preview ready!

Name	Link
🔨 Latest commit	d7f5356
🔍 Latest deploy log	https://app.netlify.com/projects/skywalking-website-preview/deploys/6ad24c489c5dee9ac0b421d4
😎 Deploy Preview	https://deploy-preview-944--skywalking-website-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[wu-sheng]

Thanks for the post, @peachisai. I checked it against apache/skywalking master, skywalking-horizon-ui main and skywalking-java, and pushed d44eac2 to this branch with the fixes below. The Netlify preview passes on it. There is one more change I'd like you to make yourself, described in the section Please update: native OTLP, no Zipkin conversion below.

Accuracy

• The English "default prompt" was a shortened version. It is now the prompt shipped in ai-evaluation.yml, which the Chinese post already had.
• Record values: SCORE and BOOLEAN are stored in PPM (0.8 becomes 800000, true becomes 1000000), not as 0.0–1.0 and 0/1. Invalid task results are skipped.
• Levels: ScoreLevelRule includes 1.0 in any rule whose max is 1.0, not in "the final range".
• The judge endpoint must also return its JSON in choices[0].message.content, not only accept the request format.
• incomplete_span also counts a missing evaluation context or trace ID, and spans missing service, provider, model, or trace reference information, not only messages. The two counters are now described with their reason values and stored meter_oap_instance_* names.
• The score metric is per model (the Virtual GenAI instance) and task, on the Models dashboard.
• Evaluation does not run "after trace processing": eligible spans are queued, and ingestion never waits for the judge.

Gaps a reader would hit

• The evaluation pages are only on Horizon UI main (the unreleased 1.1.0); 1.0.1 does not have them. The post now points to ghcr.io/apache/skywalking-horizon-ui:main.
• The Java agent does not collect Spring AI messages by default, so nothing gets evaluated. Added SW_PLUGIN_SPRINGAI_COLLECT_INPUT_MESSAGES and SW_PLUGIN_SPRINGAI_COLLECT_OUTPUT_MESSAGES.
• The trace link needs the traces:read permission.
• The shipped selector is empty, so the post now says to set SW_AI_EVALUATION=default.
• Added a note that every SCORE instruction must be written so that higher is better.

Layout and site

• data-process.png was 17874×1716 px and shrank to an unreadable strip in the post column. It is replaced by a vertical SVG in each language with the same steps, and the duplicate text flow is removed.
• Step 5's screenshots are reordered to match the text, and every image has descriptive alt text.
• The third-party relay host in the Peer field of jump-to-trace.png is pixelated.
• The directories are renamed to 2026-10-15-llm-as-judge to match date (the directory name is the URL), and the UTF-8 BOM is removed.
• Tags are now AI, Metrics, Tracing from data/blog_topics.yml. The old tags linked to /zh/tags/... pages that do not exist.

Chinese version
It is aligned with the English, which it differed from in a few added and dropped sentences, and reworded in places. "AI gateway or agent" is now "AI 网关和探针".

Please update: native OTLP, no Zipkin conversion
The post should follow the latest OTLP analysis and OTLP trace storage. OAP 11.1.0 stores OTLP traces natively by default (otlpTraceStorage: otlp) and evaluates those spans directly (AIEvaluationSpanListener#onNativeOTLPSpan). Converting OTLP spans into Zipkin form is not the path we want readers to take. Please:

1. In step 5 of both posts, remove the sentences my commit added about opening OTLP-linked records through the Zipkin view and setting SW_OTEL_TRACE_STORAGE=zipkin with receiver-zipkin and query-zipkin.
2. In step 3, state that OTLP traces are stored natively and evaluated directly, with no conversion into Zipkin form.
3. If you can, try an OTLP-instrumented app on the latest dev builds (OAP master and Horizon UI main) with the default native storage, and check that its evaluation records appear.

For context: Horizon UI main still opens an OTLP-linked record's trace through the Zipkin view (apps/ui/src/layer/traces/useResultTracePopout.ts), so under native storage the trace link does not find OTLP traces yet. That is a Horizon UI gap to fix there, not something to work around in the post. Your screenshots use a SkyWalking native trace, so they do not need to change.

Not changed in this PR

• Upstream: the shipped default task Hallucination: "Evaluate hallucination severity." is a SCORE task, but the system prompt treats 1.0 as the best result and the levels map 1.0 to excellent, so if the judge interprets higher scores as greater hallucination severity, a poor answer could be labeled excellent. Your wording ("absence of hallucinations. Higher is better.") is the right fix and should also go into oap-server/server-starter/src/main/resources/ai-evaluation.yml in apache/skywalking.

[wu-sheng]

Thanks for the update, @peachisai. The Zipkin conversion is gone, and step 3 now describes the native OTLP path. A few small things are left:

1. Chinese intro: please change "在 AI 网关或智能体中" back to "在 AI 网关和探针中". We decided to translate "agent" here as 探针.
2. Step 5 order, both posts: "like the one below" / "如下图" is now followed by the OTLP-limitation paragraph instead of the screenshot. Please move that paragraph below jump-to-trace.png.
3. Chinese step 5 sentence: "对于来自 SkyWalking 原生 Trace 的记录（如下图），有 traces:read 权限的用户可以点击右上角的 trace 链接，打开原始 Trace，查看采集到的 Agent 交互、LLM 调用和外部工具调用。" follows the English sentence structure and has four 的. Suggested wording:

   记录来自 SkyWalking 原生 Trace 时，有 traces:read 权限即可点击右上角的 trace 链接，打开原始 Trace（如下图）。在 Trace 中可以查看采集到的 Agent 交互、LLM 调用和外部工具调用。

Also, did you get a chance to try an OTLP-instrumented app with the default native storage? If its evaluation records showed up, a one-line note here is enough.

[wu-sheng]

I pushed d7f5356 with the three changes from my previous comment (the 探针 term, the step 5 paragraph order in both posts, and the reworded Chinese step 5 sentence), so you don't need to make them. The question about trying an OTLP-instrumented app with native storage still stands.