Name the store holding the span a GenAI evaluation was judged from in traceRef.type by wu-sheng · Pull Request #14152 · apache/skywalking

wu-sheng · 2026-10-19T00:02:37Z

Name the store holding the span a GenAI evaluation was judged from in `traceRef.type`

traceRef.type was the span's addressing scheme, so an OTLP reference could point into two stores: OTLP spans stored as received (otlpTraceStorage: otlp, the default), and Zipkin spans, either received by receiver-zipkin or converted from OTLP with otlpTraceStorage: zipkin. receiver-zipkin runs independently of otlpTraceStorage, so one OAP can hold judged spans in both, and a client such as Horizon's Evaluation records tab had no way to tell which store to read the trace from. Probing the datasources cannot settle it either: the store is decided per span when it arrives, and changing otlpTraceStorage later does not move earlier spans.

As suggested in review, GenAITraceRefType gains ZIPKIN, so the type itself names the store: SKYWALKING_NATIVE, OTLP for OTLP spans stored natively, or ZIPKIN. The listener path that judges a span already knows where the span went: onNativeOTLPSpan → OTLP, onZipkinSpan → ZIPKIN, the native segment listener → SKYWALKING_NATIVE. The value goes into the existing ref_type column, so no storage schema changes. OTLP and ZIPKIN share the trace ID + span ID addressing, while SKYWALKING_NATIVE uses trace ID + segment ID + span index. The relatedTrace filter in the BanyanDB, Elasticsearch and JDBC DAOs, the query validation, and the evaluation strategy's span check switch on every type and reject an unknown one. LLM-as-Judge is not in a release yet, so no stored record relies on the earlier meaning of OTLP.

Query protocol: apache/skywalking-query-protocol#178, which reverts #177, and #179, which fixes the relatedTrace comment, both merged; the submodule points at 35155de.

If this is non-trivial feature, paste the links/URLs to the design doc.
Update the documentation to include this new feature.
Tests(including UT, IT, E2E) are added to verify the new feature.
- Unit tests: each listener path records its type, ZIPKIN for a Zipkin span and OTLP for a natively stored OTLP span; a ZIPKIN reference survives the record's storage round trip.
- E2E: an evaluation-storage check in all four GenAI cases asserts traceRef.type: SKYWALKING_NATIVE (virtual-genai), ZIPKIN (otlp-virtual-genai, OTLP converted), OTLP (otlp-virtual-genai/native), and ZIPKIN (zipkin-virtual-genai, which now runs the judge so the receiver-zipkin path is covered).
- E2E: the otlp-virtual-genai evaluation span lookup is split per case, as the two cases store the span differently: the Zipkin-mode case looks the record up as ZIPKIN, the native case as OTLP.
- E2E: a new mixed-virtual-genai case runs one OAP with native OTLP storage, receiver-zipkin, the Zipkin query API and TraceQL, and judges spans from a SkyWalking-agent service, an OTLP exporter and a Zipkin exporter at the same time. For each source it checks that the newest record has the expected type, that relatedTrace finds the same span and a non-existent span finds nothing, and that only the store the type names holds the trace (SkyWalking trace query, TraceQL, Zipkin API).
- All five GenAI e2e cases passed locally on an image built from this branch: mixed-virtual-genai (4 cases), otlp-virtual-genai (24), otlp-virtual-genai/native (24), virtual-genai (28), zipkin-virtual-genai (23).
If it's UI related, attach the screenshots below.
If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
Update the CHANGES log.

Copilot

🟡 Changes recommended

Existing BanyanDB evaluation streams need additive boot reshape enabled to receive the new storage tag.

1 open issue

Existing evaluation streams miss trace_storage after upgrade · New

What changed in this PR

Records the actual span store on GenAI evaluations so clients can retrieve traces without guessing from the addressing type.

Changes:

Propagates span storage through listeners, evaluation records, and all three storage query implementations.
Adds unit tests and end-to-end checks for native, OTLP, and Zipkin ingestion.
Documents storage selection and compatibility behavior.

File	Description
test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml	Expects Zipkin storage.
test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml	Queries evaluation storage.
test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml	Enables storage verification.
test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml	Enables the evaluation judge.
test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml	Expects SkyWalking native storage.
test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml	Queries evaluation storage.
test/e2e-v2/cases/virtual-genai/e2e.yaml	Enables storage verification.
test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml	Expects native OTLP storage.
test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml	Queries evaluation storage.
test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml	Enables storage verification.
test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml	Expects converted Zipkin storage.
test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml	Queries evaluation storage.
test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml	Enables storage verification.
oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java	Reads the JDBC storage column.
oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java	Reads the Elasticsearch storage field.
oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java	Reads the BanyanDB storage tag.
oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java	Tests persistence and fallback behavior.
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/type/GenAITraceRef.java	Exposes trace storage.
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceStorage.java	Defines supported span stores.
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecord.java	Persists and resolves trace storage.
oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml	Supplies test provider rules.
oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java	Tests OTLP and Zipkin store assignment.
oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/GenAIEvaluationAnalysisListener.java	Assigns native segment storage.
oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java	Requires and persists storage metadata.
oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java	Assigns OTLP or Zipkin storage.
oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/context/AIEvaluationContext.java	Carries storage through evaluation.
docs/en/setup/backend/ai-evaluation.md	Explains storage-based trace retrieval.
docs/en/changes/changes.md	Records the feature.

_{Review effort: Balanced}

wu-sheng · 2026-10-19T02:06:24Z

Pushed 3fc0208 for two upgrade issues found in review. Both hit only storage that already holds evaluation records:

BanyanDB: @Stream(allowBootReshape = true), so an existing gen_ai_evaluation_record stream gains the trace_storage tag at boot (Copilot's comment above).
JDBC: daily tables created before the column don't get it, so the query selects NULL for it from those tables instead of failing.

The push dismissed the approvals. @wankai123 @peachisai, could you take another look?

peachisai · 2026-10-19T03:48:52Z

If we don't need to preserve compatibility with evaluation records written before this change, could we extend traceRef.type with ZIPKIN instead of introducing a separate storage field?
The values would identify the actual trace store:

SKYWALKING_NATIVE: SkyWalking-native traces.
OTLP: natively stored OTLP traces.
ZIPKIN: directly received Zipkin traces or OTLP traces converted to Zipkin.
OTLP and ZIPKIN could still share the traceId + spanId addressing logic, while the UI would use type directly to choose the query path.

… traceRef.type `traceRef.type` was the span's addressing scheme, so an `OTLP` reference could point into either the native OTLP store or the Zipkin store, and a client could not tell which one to read the trace from. GenAITraceRefType gains ZIPKIN, and each listener path records the store the span went to: OTLP for natively stored OTLP spans, ZIPKIN for spans received by receiver-zipkin or converted from OTLP with `otlpTraceStorage: zipkin`. The value goes into the existing `ref_type` column. The relatedTrace filters, the query validation and the evaluation span check switch on every type and reject an unknown one. The new mixed-virtual-genai e2e case judges spans from a SkyWalking agent, an OTLP exporter and a Zipkin exporter on one OAP, and checks that each record is found again through relatedTrace and names the only store holding its trace. The query protocol points at apache/skywalking-query-protocol#178 and #179.

wu-sheng · 2026-10-19T06:59:48Z

@peachisai Thanks, done in aeed0eb as you suggested:

GenAITraceRefType gains ZIPKIN, so traceRef.type names the store holding the judged span: SKYWALKING_NATIVE, OTLP for OTLP spans stored natively, or ZIPKIN for spans received by receiver-zipkin and OTLP spans converted with otlpTraceStorage: zipkin. onZipkinSpan records ZIPKIN and onNativeOTLPSpan records OTLP.
The type is stored in the existing ref_type column, so the separate storage field and trace_storage column are gone, and with them the allowBootReshape opt-in and the JDBC column check from 3fc0208. LLM-as-Judge is not in a release yet, so no stored record relies on the earlier meaning of OTLP.
OTLP and ZIPKIN share the trace ID + span ID addressing, and SKYWALKING_NATIVE keeps trace ID + segment ID + span index. The relatedTrace filters in the BanyanDB, Elasticsearch and JDBC DAOs, the query validation, and the evaluation span check switch on every type and reject an unknown one.
Query protocol: Name the Zipkin store as a GenAITraceRefType value, replacing GenAITraceStorage skywalking-query-protocol#178 reverts modify the expose port of collector image #177 and adds ZIPKIN, and fix functional issues after update new package name #179 fixes the relatedTrace comment; the submodule points at 35155de.
E2E: the four evaluation-storage checks assert traceRef.type, and the otlp-virtual-genai span lookup is split per case, looking the record up as ZIPKIN in Zipkin mode and as OTLP in the native variant. A new mixed-virtual-genai case runs one OAP that judges spans from a SkyWalking agent, an OTLP exporter and a Zipkin exporter at the same time; for each type it finds the record again through relatedTrace and checks that only the store the type names holds its trace.

Horizon UI needs a follow-up to open ZIPKIN references in the Zipkin view and OTLP ones in the OTLP view.

@wankai123 @peachisai, could you take another look?

…d names OAP's evaluation traceRef.type now names the store holding the judged span (apache/skywalking#14152): SKYWALKING_NATIVE, OTLP (stored as OTLP, read through TraceQL's OTLP datasource) or ZIPKIN (the Zipkin receiver, or OTLP converted with otlpTraceStorage: zipkin). The Evaluation records tab follows it: a native trace opens on the native waterfall, an OTLP one on the TraceQL popout with the judged span selected, a Zipkin one on the Zipkin popout. Pick span... reads the same store, and the Trace store condition offers all three, matched exactly by OAP. A TraceQL trace read tells a missing store from a missing trace: a source with no TraceQL URL, and a 404 on the datasource path (a datasource OAP does not enable, or a context path Horizon has wrong), each say what to set. The TraceQL popout re-reads a trace still landing, as the native and Zipkin popouts do, and a view reused for another trace gets a fresh retry budget. e2e: OAP is pinned at the #14152 merge. The genai case adds TraceQL with the OTLP datasource, the Zipkin receiver and a caller exporting Zipkin, with readiness checks and specs for the OTLP and Zipkin stores. The core endpoint specs pick the fixture's endpoint by name: OAP now lists endpoints known only from logs or cross-thread spans (#14122), and those carry no endpoint metrics.

wu-sheng mentioned this pull request Oct 19, 2026

Name the store holding the span a GenAI evaluation was produced from apache/skywalking-query-protocol#177

Merged

wu-sheng requested a balanced review from Copilot October 19, 2026 00:03

Copilot started reviewing on behalf of wu-sheng October 19, 2026 00:03

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

Comment thread .../java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecord.java Outdated

wu-sheng added this to the 11.1.0 milestone Oct 19, 2026

wu-sheng added backend OAP backend related. enhancement Enhancement on performance or codes labels Oct 19, 2026

wu-sheng requested review from peachisai and wankai123 October 19, 2026 00:17

wankai123 previously approved these changes Oct 19, 2026

View reviewed changes

peachisai previously approved these changes Oct 19, 2026

View reviewed changes

wu-sheng dismissed stale reviews from peachisai and wankai123 via 3fc0208 October 19, 2026 01:54

wu-sheng mentioned this pull request Oct 19, 2026

Name the Zipkin store as a GenAITraceRefType value, replacing GenAITraceStorage apache/skywalking-query-protocol#178

Merged

wu-sheng force-pushed the genai-evaluation-trace-storage branch from 3fc0208 to b8ef2e3 Compare October 19, 2026 05:12

wu-sheng changed the title ~~Record which store holds the span a GenAI evaluation was judged from~~ Name the store holding the span a GenAI evaluation was judged from in traceRef.type Oct 19, 2026

wu-sheng mentioned this pull request Oct 19, 2026

Name all three trace stores in the relatedTrace comment apache/skywalking-query-protocol#179

Merged

wu-sheng force-pushed the genai-evaluation-trace-storage branch from b8ef2e3 to aeed0eb Compare October 19, 2026 06:59

wu-sheng requested review from peachisai and wankai123 October 19, 2026 07:09

peachisai approved these changes Oct 19, 2026

View reviewed changes

wu-sheng mentioned this pull request Oct 19, 2026

Fix Zabbix agent sample timestamps dropping the sub-second part. #14151

Merged

2 tasks done

wu-sheng merged commit f34b9c7 into master Oct 19, 2026
478 of 483 checks passed

wu-sheng deleted the genai-evaluation-trace-storage branch October 19, 2026 23:04

wu-sheng mentioned this pull request Oct 20, 2026

feat(evaluation): open a judged span's trace from the store its record names apache/skywalking-horizon-ui#204

Merged

Navigation Menu

Repository navigation

Name the store holding the span a GenAI evaluation was judged from in traceRef.type - #14152

Name the store holding the span a GenAI evaluation was judged from in traceRef.type#14152
wu-sheng merged 1 commit into
masterfrom
genai-evaluation-trace-storage

wu-sheng commented Oct 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

wu-sheng commented Oct 19, 2026

Uh oh!

peachisai commented Oct 19, 2026

Uh oh!

wu-sheng commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

wu-sheng commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!