Name the store holding the span a GenAI evaluation was judged from in traceRef.type

.github/workflows/skywalking.yaml

@@ -707,6 +707,8 @@ jobs:
             config: test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml
           - name: Zipkin Virtual GenAI
             config: test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml
+          - name: Mixed Virtual GenAI
+            config: test/e2e-v2/cases/mixed-virtual-genai/e2e.yaml
           - name: iOS Monitoring
             config: test/e2e-v2/cases/ios/e2e.yaml
           - name: WeChat Mini Program Monitoring

docs/en/changes/changes.md

@@ -56,6 +56,7 @@
 * Fix the Envoy ALS sidecar internal request and response latencies dropping whole seconds: only the `nanos` part of each protobuf `Duration` was read. The internal response latency is now left out of the internal latency metrics when the access log lacks the upstream or downstream timing it is computed from, such as when Envoy answers an upstream timeout with its own 504, instead of counting the whole wait or going negative.
 * Sync the Envoy access log proto, `envoy/data/accesslog/v3/accesslog.proto`, with Envoy v1.40.0, adding the `envoy/annotations/deprecation.proto` it imports. With it, the Envoy ALS sidecar internal latencies leave out what the access log cannot measure instead of counting it: the internal request latency of an HTTP request Envoy retried or hedged, or of a TCP connection that took more than one connect attempt (`upstream_request_attempt_count` above 1), as Envoy measures the upstream timings from the start of the downstream request or connection, so the time to the first upstream byte can include the earlier attempts and the delays between them; and the internal response latency when the response details are set to anything but `via_upstream`, as for Envoy's own 504 of a timeout during a retry backoff, which can carry an earlier attempt's upstream timing, and, conservatively, for an upstream response that a later reset or timeout ended. The adapters report such a latency as `-1`, and the `*_sidecar_internal_req_latency_nanos` and `*_sidecar_internal_resp_latency_nanos` metrics in `mesh.oal` filter it out with `filter(... >= 0)`. A customized `mesh.oal` needs the same filters, or the `-1` values are averaged in.
 * Fix OAL `>`, `>=`, `<` and `<=` filters on a `long` or `double` source field, and `==` on a `long` one, failing to compile when they compare it with a number, such as `filter(sideCar.internalRequestLatencyNanos >= 0)`. The generated code passed the number as an `int`, and the comparison matchers have no `match(long, int)` or `match(double, int)` overload, as Javassist does not widen an argument. The number is now written in the field's type, `0L` or `0.0`; filters on `int` fields, which already worked, generate the same code.
+* Add `ZIPKIN` to the GenAI evaluation record's `traceRef.type`, for spans received by `receiver-zipkin` and OTLP spans converted with `otlpTraceStorage: zipkin`, so the type names the store holding the judged span: `SKYWALKING_NATIVE`, `OTLP` for OTLP spans stored natively, or `ZIPKIN`. Zipkin and OTLP spans are both addressed by span ID, and one OAP can hold both, so an `OTLP` reference alone could not tell a client which store to read the trace from. The type is decided when the span is received, so records written under different `otlpTraceStorage` settings each name their own store.
 
 #### UI
 * Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records.

docs/en/setup/backend/ai-evaluation.md

@@ -98,6 +98,16 @@ Evaluation records are available through the `queryGenAIEvaluationRecord` GraphQ
 
 The metric value is PPM. Divide it by `1000000` when displaying the original score.
 
+The trace reference's `type` names the store holding the judged span, so a client reads the trace from the right place:
+
+| `type` | Spans | Addressed by | Read through |
+|---|---|---|---|
+| `SKYWALKING_NATIVE` | Received from SkyWalking agents | `traceId`, `segmentId` and `spanIndex` | The SkyWalking trace query |
+| `OTLP` | Received over OTLP with `receiver-otel` `otlpTraceStorage: otlp`, the default | `traceId` and `spanId` | The TraceQL OTLP datasource |
+| `ZIPKIN` | Received by `receiver-zipkin`, or over OTLP with `otlpTraceStorage: zipkin` | `traceId` and `spanId` | The Zipkin query API |
+
+One OAP can hold both `OTLP` and `ZIPKIN` spans at once, since `receiver-zipkin` runs independently of `otlpTraceStorage`. The type is decided when a span is received, so changing `otlpTraceStorage` later does not change the records already written.
+
 ## Operational considerations
 
 Evaluation runs asynchronously after a GenAI span is sampled. A slow or unavailable judge affects evaluation throughput, not trace ingestion, but it can fill the local queue and increase dropped-evaluation counters. Monitor the `ai_evaluation_dropped_count` and `ai_evaluation_error_count` telemetry metrics when tuning the queue or judge settings.

oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java

@@ -77,7 +77,8 @@ public SpanListenerResult onZipkinSpan(final ZipkinSpan span) {
         if (!shouldSample(span.getTraceId())) {
             return SpanListenerResult.CONTINUE;
         }
-        sample(span.getTraceId(),
+        sample(GenAITraceRefType.ZIPKIN,
+                span.getTraceId(),
                 span.getSpanId(),
                 span.getLocalEndpointServiceName(),
                 span.getName(),
@@ -116,7 +117,8 @@ public SpanListenerResult onNativeOTLPSpan(final OTLPSpanReader span,
         if (!shouldSample(span.traceId())) {
             return SpanListenerResult.CONTINUE;
         }
-        sample(span.traceId(),
+        sample(GenAITraceRefType.OTLP,
+                span.traceId(),
                 span.spanId(),
                 resolveServiceName(resourceAttributes),
                 span.spanName(),
@@ -141,7 +143,8 @@ private static String resolveServiceName(final Map<String, String> resourceAttri
         return "";
     }
 
-    private void sample(final String traceId,
+    private void sample(final GenAITraceRefType traceRefType,
+                        final String traceId,
                         final String spanId,
                         final String serviceName,
                         final String operationName,
@@ -152,7 +155,7 @@ private void sample(final String traceId,
         final GenAIContextResolver.Result genAIContext = GenAIContextResolver.resolve(tags);
         evaluationService.sample(AIEvaluationContext.builder()
                 .traceId(traceId)
-                .traceRefType(GenAITraceRefType.OTLP)
+                .traceRefType(traceRefType)
                 .spanId(spanId)
                 .serviceName(serviceName)
                 .operationName(operationName)

oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java

@@ -178,13 +178,20 @@ private static boolean completeLLMCallSpan(final AIEvaluationContext context) {
             return false;
         }
 
-        if (context.getTraceRefType() == GenAITraceRefType.SKYWALKING_NATIVE) {
-            if (StringUtil.isBlank(context.getSegmentId()) || context.getSpanIndex() == null) {
+        switch (context.getTraceRefType()) {
+            case SKYWALKING_NATIVE:
+                if (StringUtil.isBlank(context.getSegmentId()) || context.getSpanIndex() == null) {
+                    return false;
+                }
+                break;
+            case OTLP:
+            case ZIPKIN:
+                if (StringUtil.isBlank(context.getSpanId())) {
+                    return false;
+                }
+                break;
+            default:
                 return false;
-            }
-        } else if (context.getTraceRefType() != GenAITraceRefType.OTLP
-            || StringUtil.isBlank(context.getSpanId())) {
-            return false;
         }
 
         final String inputMessages = context.getTags().get(GenAISemanticAttributes.INPUT_MESSAGES);

oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java

@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.ai.evaluation.listener;
+
+import com.google.gson.JsonObject;
+import java.util.Collections;
+import java.util.Map;
+import org.apache.skywalking.oap.server.ai.evaluation.AIEvaluationModule;
+import org.apache.skywalking.oap.server.ai.evaluation.context.AIEvaluationContext;
+import org.apache.skywalking.oap.server.ai.evaluation.service.IAIEvaluationService;
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.trace.OTLPSpanReader;
+import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinSpan;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder;
+import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder;
+import org.apache.skywalking.oap.server.library.util.genai.GenAISemanticAttributes;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+class AIEvaluationSpanListenerTest {
+    private static final String TRACE_ID = "4bf92f3577b34da6a3ce929d0e0e4736";
+    private static final String SPAN_ID = "00f067aa0ba902b7";
+    private static final String MODEL = "gpt-4.1-mini";
+
+    private IAIEvaluationService evaluationService;
+    private AIEvaluationSpanListener listener;
+
+    @BeforeEach
+    void setUp() {
+        evaluationService = mock(IAIEvaluationService.class);
+        when(evaluationService.shouldSample(anyString())).thenReturn(true);
+        final ModuleServiceHolder provider = mock(ModuleServiceHolder.class);
+        when(provider.getService(IAIEvaluationService.class)).thenReturn(evaluationService);
+        final ModuleProviderHolder module = mock(ModuleProviderHolder.class);
+        when(module.provider()).thenReturn(provider);
+        final ModuleManager moduleManager = mock(ModuleManager.class);
+        when(moduleManager.find(AIEvaluationModule.NAME)).thenReturn(module);
+        listener = new AIEvaluationSpanListener();
+        listener.init(moduleManager);
+    }
+
+    @Test
+    void recordsAZipkinSpanAsStoredInZipkin() {
+        final JsonObject tags = new JsonObject();
+        tags.addProperty(GenAISemanticAttributes.RESPONSE_MODEL, MODEL);
+        final ZipkinSpan span = new ZipkinSpan();
+        span.setTraceId(TRACE_ID);
+        span.setSpanId(SPAN_ID);
+        span.setTags(tags);
+
+        listener.onZipkinSpan(span);
+
+        final AIEvaluationContext context = sampledContext();
+        assertEquals(GenAITraceRefType.ZIPKIN, context.getTraceRefType());
+        assertEquals(SPAN_ID, context.getSpanId());
+    }
+
+    @Test
+    void recordsANativeOtlpSpanAsStoredInOtlp() {
+        final OTLPSpanReader span = new OTLPSpanReader() {
+            @Override
+            public String traceId() {
+                return TRACE_ID;
+            }
+
+            @Override
+            public String spanId() {
+                return SPAN_ID;
+            }
+
+            @Override
+            public String spanName() {
+                return "chat " + MODEL;
+            }
+
+            @Override
+            public String spanKind() {
+                return "SPAN_KIND_CLIENT";
+            }
+
+            @Override
+            public long startTimeNanos() {
+                return 0;
+            }
+
+            @Override
+            public long endTimeNanos() {
+                return 0;
+            }
+
+            @Override
+            public String getAttribute(final String key) {
+                return attributes().get(key);
+            }
+
+            @Override
+            public Map<String, String> attributes() {
+                return Collections.singletonMap(GenAISemanticAttributes.RESPONSE_MODEL, MODEL);
+            }
+        };
+
+        listener.onNativeOTLPSpan(span, Collections.emptyMap(), "opentelemetry.instrumentation.openai_v2", "2.0");
+
+        final AIEvaluationContext context = sampledContext();
+        assertEquals(GenAITraceRefType.OTLP, context.getTraceRefType());
+        assertEquals(SPAN_ID, context.getSpanId());
+    }
+
+    private AIEvaluationContext sampledContext() {
+        final ArgumentCaptor<AIEvaluationContext> captor = ArgumentCaptor.forClass(AIEvaluationContext.class);
+        verify(evaluationService).sample(captor.capture());
+        return captor.getValue();
+    }
+}

oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml

@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# The span listeners resolve the provider of a GenAI span from these rules.
+providers:
+  - provider: openai
+    prefix-match:
+      - gpt
+    models:
+      - name: gpt-4.1-mini
+        input-estimated-cost-per-m: 0.4
+        output-estimated-cost-per-m: 1.6

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/GenAIEvaluationRecordQueryService.java

@@ -20,7 +20,6 @@
 
 import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationRecordSortBy;
 import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationValueType;
-import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
 import org.apache.skywalking.oap.server.core.query.enumeration.Order;
 import org.apache.skywalking.oap.server.core.query.input.Duration;
 import org.apache.skywalking.oap.server.core.query.input.GenAITraceScopeCondition;
@@ -148,17 +147,24 @@ private static void validateRelatedTrace(final GenAITraceScopeCondition relatedT
         }
         final boolean hasSegmentId = relatedTrace.getSegmentId() != null && !relatedTrace.getSegmentId().isEmpty();
         final boolean hasSpanId = relatedTrace.getSpanId() != null && !relatedTrace.getSpanId().isEmpty();
-        if (GenAITraceRefType.SKYWALKING_NATIVE.equals(relatedTrace.getType())) {
-            if (hasSpanId) {
-                throw new IllegalArgumentException("Native relatedTrace cannot contain spanId");
-            }
-            if (relatedTrace.getSpanIndex() != null && !hasSegmentId) {
-                throw new IllegalArgumentException("Native relatedTrace.spanIndex requires segmentId");
-            }
-        } else if (GenAITraceRefType.OTLP.equals(relatedTrace.getType())) {
-            if (hasSegmentId || relatedTrace.getSpanIndex() != null) {
-                throw new IllegalArgumentException("OTLP relatedTrace cannot contain segmentId or spanIndex");
-            }
+        switch (relatedTrace.getType()) {
+            case SKYWALKING_NATIVE:
+                if (hasSpanId) {
+                    throw new IllegalArgumentException("Native relatedTrace cannot contain spanId");
+                }
+                if (relatedTrace.getSpanIndex() != null && !hasSegmentId) {
+                    throw new IllegalArgumentException("Native relatedTrace.spanIndex requires segmentId");
+                }
+                break;
+            case OTLP:
+            case ZIPKIN:
+                if (hasSegmentId || relatedTrace.getSpanIndex() != null) {
+                    throw new IllegalArgumentException(
+                        relatedTrace.getType() + " relatedTrace cannot contain segmentId or spanIndex");
+                }
+                break;
+            default:
+                throw new IllegalArgumentException("Unsupported relatedTrace.type: " + relatedTrace.getType());
         }
     }
 

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceRefType.java

@@ -18,7 +18,20 @@
 
 package org.apache.skywalking.oap.server.core.query.enumeration;
 
+/**
+ * The store holding the span a GenAI evaluation was produced from, which also sets how the span is addressed.
+ */
 public enum GenAITraceRefType {
+    /**
+     * SkyWalking native segments, received from SkyWalking agents.
+     */
     SKYWALKING_NATIVE,
-    OTLP
+    /**
+     * OTLP spans stored as received, with receiver-otel {@code otlpTraceStorage: otlp}.
+     */
+    OTLP,
+    /**
+     * Spans received by receiver-zipkin, or OTLP spans converted with receiver-otel {@code otlpTraceStorage: zipkin}.
+     */
+    ZIPKIN
 }

oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java

@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.manual.genai;
+
+import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType;
+import org.apache.skywalking.oap.server.core.query.type.GenAITraceRef;
+import org.apache.skywalking.oap.server.core.storage.type.HashMapConverter;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+class GenAIEvaluationRecordTest {
+
+    @Test
+    void keepsAZipkinTraceRefThroughStorage() {
+        final GenAIEvaluationRecord record = GenAIEvaluationRecord.create("openai", "gpt-4.1-mini");
+        record.setUniqueId("unique");
+        record.setTraceId("trace");
+        record.setRefType(GenAITraceRefType.ZIPKIN.name());
+        record.setSpanId("0123456789abcdef");
+        record.setEvaluationTime(1_790_000_000_000L);
+        record.setTimeBucket(20261019000000L);
+
+        final GenAIEvaluationRecord.Builder builder = new GenAIEvaluationRecord.Builder();
+        final HashMapConverter.ToStorage toStorage = new HashMapConverter.ToStorage();
+        builder.entity2Storage(record, toStorage);
+        final GenAITraceRef ref = builder.storage2Entity(new HashMapConverter.ToEntity(toStorage.obtain()))
+                                         .getTraceRef();
+
+        assertEquals(GenAITraceRefType.ZIPKIN, ref.getType());
+        assertEquals("0123456789abcdef", ref.getSpanId());
+    }
+}