diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml index 257f74494ec5..4628e2c51697 100644 --- a/.github/workflows/skywalking.yaml +++ b/.github/workflows/skywalking.yaml @@ -707,6 +707,8 @@ jobs: config: test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml - name: Zipkin Virtual GenAI config: test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml + - name: Mixed Virtual GenAI + config: test/e2e-v2/cases/mixed-virtual-genai/e2e.yaml - name: iOS Monitoring config: test/e2e-v2/cases/ios/e2e.yaml - name: WeChat Mini Program Monitoring diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 8bc2676a288c..0fa42226a924 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -56,6 +56,7 @@ * Fix the Envoy ALS sidecar internal request and response latencies dropping whole seconds: only the `nanos` part of each protobuf `Duration` was read. The internal response latency is now left out of the internal latency metrics when the access log lacks the upstream or downstream timing it is computed from, such as when Envoy answers an upstream timeout with its own 504, instead of counting the whole wait or going negative. * Sync the Envoy access log proto, `envoy/data/accesslog/v3/accesslog.proto`, with Envoy v1.40.0, adding the `envoy/annotations/deprecation.proto` it imports. With it, the Envoy ALS sidecar internal latencies leave out what the access log cannot measure instead of counting it: the internal request latency of an HTTP request Envoy retried or hedged, or of a TCP connection that took more than one connect attempt (`upstream_request_attempt_count` above 1), as Envoy measures the upstream timings from the start of the downstream request or connection, so the time to the first upstream byte can include the earlier attempts and the delays between them; and the internal response latency when the response details are set to anything but `via_upstream`, as for Envoy's own 504 of a timeout during a retry backoff, which can carry an earlier attempt's upstream timing, and, conservatively, for an upstream response that a later reset or timeout ended. The adapters report such a latency as `-1`, and the `*_sidecar_internal_req_latency_nanos` and `*_sidecar_internal_resp_latency_nanos` metrics in `mesh.oal` filter it out with `filter(... >= 0)`. A customized `mesh.oal` needs the same filters, or the `-1` values are averaged in. * Fix OAL `>`, `>=`, `<` and `<=` filters on a `long` or `double` source field, and `==` on a `long` one, failing to compile when they compare it with a number, such as `filter(sideCar.internalRequestLatencyNanos >= 0)`. The generated code passed the number as an `int`, and the comparison matchers have no `match(long, int)` or `match(double, int)` overload, as Javassist does not widen an argument. The number is now written in the field's type, `0L` or `0.0`; filters on `int` fields, which already worked, generate the same code. +* Add `ZIPKIN` to the GenAI evaluation record's `traceRef.type`, for spans received by `receiver-zipkin` and OTLP spans converted with `otlpTraceStorage: zipkin`, so the type names the store holding the judged span: `SKYWALKING_NATIVE`, `OTLP` for OTLP spans stored natively, or `ZIPKIN`. Zipkin and OTLP spans are both addressed by span ID, and one OAP can hold both, so an `OTLP` reference alone could not tell a client which store to read the trace from. The type is decided when the span is received, so records written under different `otlpTraceStorage` settings each name their own store. #### UI * Add a Virtual GenAI evaluation-record page and evaluation-score chart in Horizon UI, so operators can inspect evaluation result, level, reason, judge model, timestamp, trace linkage, and the `gen_ai_model_evaluation_score_ppm` trend for evaluated records. diff --git a/docs/en/setup/backend/ai-evaluation.md b/docs/en/setup/backend/ai-evaluation.md index 13b8a956c90b..700e38eb8061 100644 --- a/docs/en/setup/backend/ai-evaluation.md +++ b/docs/en/setup/backend/ai-evaluation.md @@ -98,6 +98,16 @@ Evaluation records are available through the `queryGenAIEvaluationRecord` GraphQ The metric value is PPM. Divide it by `1000000` when displaying the original score. +The trace reference's `type` names the store holding the judged span, so a client reads the trace from the right place: + +| `type` | Spans | Addressed by | Read through | +|---|---|---|---| +| `SKYWALKING_NATIVE` | Received from SkyWalking agents | `traceId`, `segmentId` and `spanIndex` | The SkyWalking trace query | +| `OTLP` | Received over OTLP with `receiver-otel` `otlpTraceStorage: otlp`, the default | `traceId` and `spanId` | The TraceQL OTLP datasource | +| `ZIPKIN` | Received by `receiver-zipkin`, or over OTLP with `otlpTraceStorage: zipkin` | `traceId` and `spanId` | The Zipkin query API | + +One OAP can hold both `OTLP` and `ZIPKIN` spans at once, since `receiver-zipkin` runs independently of `otlpTraceStorage`. The type is decided when a span is received, so changing `otlpTraceStorage` later does not change the records already written. + ## Operational considerations Evaluation runs asynchronously after a GenAI span is sampled. A slow or unavailable judge affects evaluation throughput, not trace ingestion, but it can fill the local queue and increase dropped-evaluation counters. Monitor the `ai_evaluation_dropped_count` and `ai_evaluation_error_count` telemetry metrics when tuning the queue or judge settings. diff --git a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java index 1bc9afc88344..c90802df8286 100644 --- a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java +++ b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListener.java @@ -77,7 +77,8 @@ public SpanListenerResult onZipkinSpan(final ZipkinSpan span) { if (!shouldSample(span.getTraceId())) { return SpanListenerResult.CONTINUE; } - sample(span.getTraceId(), + sample(GenAITraceRefType.ZIPKIN, + span.getTraceId(), span.getSpanId(), span.getLocalEndpointServiceName(), span.getName(), @@ -116,7 +117,8 @@ public SpanListenerResult onNativeOTLPSpan(final OTLPSpanReader span, if (!shouldSample(span.traceId())) { return SpanListenerResult.CONTINUE; } - sample(span.traceId(), + sample(GenAITraceRefType.OTLP, + span.traceId(), span.spanId(), resolveServiceName(resourceAttributes), span.spanName(), @@ -141,7 +143,8 @@ private static String resolveServiceName(final Map resourceAttri return ""; } - private void sample(final String traceId, + private void sample(final GenAITraceRefType traceRefType, + final String traceId, final String spanId, final String serviceName, final String operationName, @@ -152,7 +155,7 @@ private void sample(final String traceId, final GenAIContextResolver.Result genAIContext = GenAIContextResolver.resolve(tags); evaluationService.sample(AIEvaluationContext.builder() .traceId(traceId) - .traceRefType(GenAITraceRefType.OTLP) + .traceRefType(traceRefType) .spanId(spanId) .serviceName(serviceName) .operationName(operationName) diff --git a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java index 1dbf417417ef..7ec3acd58f59 100644 --- a/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java +++ b/oap-server/analyzer/ai-evaluation/src/main/java/org/apache/skywalking/oap/server/ai/evaluation/service/strategy/span/SpanAIEvaluationStrategy.java @@ -178,13 +178,20 @@ private static boolean completeLLMCallSpan(final AIEvaluationContext context) { return false; } - if (context.getTraceRefType() == GenAITraceRefType.SKYWALKING_NATIVE) { - if (StringUtil.isBlank(context.getSegmentId()) || context.getSpanIndex() == null) { + switch (context.getTraceRefType()) { + case SKYWALKING_NATIVE: + if (StringUtil.isBlank(context.getSegmentId()) || context.getSpanIndex() == null) { + return false; + } + break; + case OTLP: + case ZIPKIN: + if (StringUtil.isBlank(context.getSpanId())) { + return false; + } + break; + default: return false; - } - } else if (context.getTraceRefType() != GenAITraceRefType.OTLP - || StringUtil.isBlank(context.getSpanId())) { - return false; } final String inputMessages = context.getTags().get(GenAISemanticAttributes.INPUT_MESSAGES); diff --git a/oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java b/oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java new file mode 100644 index 000000000000..2709d62f4e66 --- /dev/null +++ b/oap-server/analyzer/ai-evaluation/src/test/java/org/apache/skywalking/oap/server/ai/evaluation/listener/AIEvaluationSpanListenerTest.java @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.ai.evaluation.listener; + +import com.google.gson.JsonObject; +import java.util.Collections; +import java.util.Map; +import org.apache.skywalking.oap.server.ai.evaluation.AIEvaluationModule; +import org.apache.skywalking.oap.server.ai.evaluation.context.AIEvaluationContext; +import org.apache.skywalking.oap.server.ai.evaluation.service.IAIEvaluationService; +import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType; +import org.apache.skywalking.oap.server.core.trace.OTLPSpanReader; +import org.apache.skywalking.oap.server.core.zipkin.source.ZipkinSpan; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder; +import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder; +import org.apache.skywalking.oap.server.library.util.genai.GenAISemanticAttributes; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class AIEvaluationSpanListenerTest { + private static final String TRACE_ID = "4bf92f3577b34da6a3ce929d0e0e4736"; + private static final String SPAN_ID = "00f067aa0ba902b7"; + private static final String MODEL = "gpt-4.1-mini"; + + private IAIEvaluationService evaluationService; + private AIEvaluationSpanListener listener; + + @BeforeEach + void setUp() { + evaluationService = mock(IAIEvaluationService.class); + when(evaluationService.shouldSample(anyString())).thenReturn(true); + final ModuleServiceHolder provider = mock(ModuleServiceHolder.class); + when(provider.getService(IAIEvaluationService.class)).thenReturn(evaluationService); + final ModuleProviderHolder module = mock(ModuleProviderHolder.class); + when(module.provider()).thenReturn(provider); + final ModuleManager moduleManager = mock(ModuleManager.class); + when(moduleManager.find(AIEvaluationModule.NAME)).thenReturn(module); + listener = new AIEvaluationSpanListener(); + listener.init(moduleManager); + } + + @Test + void recordsAZipkinSpanAsStoredInZipkin() { + final JsonObject tags = new JsonObject(); + tags.addProperty(GenAISemanticAttributes.RESPONSE_MODEL, MODEL); + final ZipkinSpan span = new ZipkinSpan(); + span.setTraceId(TRACE_ID); + span.setSpanId(SPAN_ID); + span.setTags(tags); + + listener.onZipkinSpan(span); + + final AIEvaluationContext context = sampledContext(); + assertEquals(GenAITraceRefType.ZIPKIN, context.getTraceRefType()); + assertEquals(SPAN_ID, context.getSpanId()); + } + + @Test + void recordsANativeOtlpSpanAsStoredInOtlp() { + final OTLPSpanReader span = new OTLPSpanReader() { + @Override + public String traceId() { + return TRACE_ID; + } + + @Override + public String spanId() { + return SPAN_ID; + } + + @Override + public String spanName() { + return "chat " + MODEL; + } + + @Override + public String spanKind() { + return "SPAN_KIND_CLIENT"; + } + + @Override + public long startTimeNanos() { + return 0; + } + + @Override + public long endTimeNanos() { + return 0; + } + + @Override + public String getAttribute(final String key) { + return attributes().get(key); + } + + @Override + public Map attributes() { + return Collections.singletonMap(GenAISemanticAttributes.RESPONSE_MODEL, MODEL); + } + }; + + listener.onNativeOTLPSpan(span, Collections.emptyMap(), "opentelemetry.instrumentation.openai_v2", "2.0"); + + final AIEvaluationContext context = sampledContext(); + assertEquals(GenAITraceRefType.OTLP, context.getTraceRefType()); + assertEquals(SPAN_ID, context.getSpanId()); + } + + private AIEvaluationContext sampledContext() { + final ArgumentCaptor captor = ArgumentCaptor.forClass(AIEvaluationContext.class); + verify(evaluationService).sample(captor.capture()); + return captor.getValue(); + } +} diff --git a/oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml b/oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml new file mode 100644 index 000000000000..420a8c65f4da --- /dev/null +++ b/oap-server/analyzer/ai-evaluation/src/test/resources/gen-ai-config.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The span listeners resolve the provider of a GenAI span from these rules. +providers: + - provider: openai + prefix-match: + - gpt + models: + - name: gpt-4.1-mini + input-estimated-cost-per-m: 0.4 + output-estimated-cost-per-m: 1.6 diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/GenAIEvaluationRecordQueryService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/GenAIEvaluationRecordQueryService.java index 5733c6108946..e4c455742fcd 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/GenAIEvaluationRecordQueryService.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/GenAIEvaluationRecordQueryService.java @@ -20,7 +20,6 @@ import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationRecordSortBy; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationValueType; -import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType; import org.apache.skywalking.oap.server.core.query.enumeration.Order; import org.apache.skywalking.oap.server.core.query.input.Duration; import org.apache.skywalking.oap.server.core.query.input.GenAITraceScopeCondition; @@ -148,17 +147,24 @@ private static void validateRelatedTrace(final GenAITraceScopeCondition relatedT } final boolean hasSegmentId = relatedTrace.getSegmentId() != null && !relatedTrace.getSegmentId().isEmpty(); final boolean hasSpanId = relatedTrace.getSpanId() != null && !relatedTrace.getSpanId().isEmpty(); - if (GenAITraceRefType.SKYWALKING_NATIVE.equals(relatedTrace.getType())) { - if (hasSpanId) { - throw new IllegalArgumentException("Native relatedTrace cannot contain spanId"); - } - if (relatedTrace.getSpanIndex() != null && !hasSegmentId) { - throw new IllegalArgumentException("Native relatedTrace.spanIndex requires segmentId"); - } - } else if (GenAITraceRefType.OTLP.equals(relatedTrace.getType())) { - if (hasSegmentId || relatedTrace.getSpanIndex() != null) { - throw new IllegalArgumentException("OTLP relatedTrace cannot contain segmentId or spanIndex"); - } + switch (relatedTrace.getType()) { + case SKYWALKING_NATIVE: + if (hasSpanId) { + throw new IllegalArgumentException("Native relatedTrace cannot contain spanId"); + } + if (relatedTrace.getSpanIndex() != null && !hasSegmentId) { + throw new IllegalArgumentException("Native relatedTrace.spanIndex requires segmentId"); + } + break; + case OTLP: + case ZIPKIN: + if (hasSegmentId || relatedTrace.getSpanIndex() != null) { + throw new IllegalArgumentException( + relatedTrace.getType() + " relatedTrace cannot contain segmentId or spanIndex"); + } + break; + default: + throw new IllegalArgumentException("Unsupported relatedTrace.type: " + relatedTrace.getType()); } } diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceRefType.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceRefType.java index d0affc0ddf79..5392eff7ceba 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceRefType.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/enumeration/GenAITraceRefType.java @@ -18,7 +18,20 @@ package org.apache.skywalking.oap.server.core.query.enumeration; +/** + * The store holding the span a GenAI evaluation was produced from, which also sets how the span is addressed. + */ public enum GenAITraceRefType { + /** + * SkyWalking native segments, received from SkyWalking agents. + */ SKYWALKING_NATIVE, - OTLP + /** + * OTLP spans stored as received, with receiver-otel {@code otlpTraceStorage: otlp}. + */ + OTLP, + /** + * Spans received by receiver-zipkin, or OTLP spans converted with receiver-otel {@code otlpTraceStorage: zipkin}. + */ + ZIPKIN } diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java new file mode 100644 index 000000000000..b41fdeb360e7 --- /dev/null +++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/manual/genai/GenAIEvaluationRecordTest.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.core.analysis.manual.genai; + +import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType; +import org.apache.skywalking.oap.server.core.query.type.GenAITraceRef; +import org.apache.skywalking.oap.server.core.storage.type.HashMapConverter; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +class GenAIEvaluationRecordTest { + + @Test + void keepsAZipkinTraceRefThroughStorage() { + final GenAIEvaluationRecord record = GenAIEvaluationRecord.create("openai", "gpt-4.1-mini"); + record.setUniqueId("unique"); + record.setTraceId("trace"); + record.setRefType(GenAITraceRefType.ZIPKIN.name()); + record.setSpanId("0123456789abcdef"); + record.setEvaluationTime(1_790_000_000_000L); + record.setTimeBucket(20261019000000L); + + final GenAIEvaluationRecord.Builder builder = new GenAIEvaluationRecord.Builder(); + final HashMapConverter.ToStorage toStorage = new HashMapConverter.ToStorage(); + builder.entity2Storage(record, toStorage); + final GenAITraceRef ref = builder.storage2Entity(new HashMapConverter.ToEntity(toStorage.obtain())) + .getTraceRef(); + + assertEquals(GenAITraceRefType.ZIPKIN, ref.getType()); + assertEquals("0123456789abcdef", ref.getSpanId()); + } +} diff --git a/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol b/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol index 689ddfea3e7c..35155de7564a 160000 --- a/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol +++ b/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol @@ -1 +1 @@ -Subproject commit 689ddfea3e7ccf32400a5be34a0538a013d23eb9 +Subproject commit 35155de7564af962144e662020700cedcdf2ab9d diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java index 15d2e01062b1..c627d29577d6 100644 --- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java +++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/stream/BanyanDBGenAIEvaluationRecordQueryDAO.java @@ -25,7 +25,6 @@ import org.apache.skywalking.oap.server.core.analysis.manual.genai.GenAIEvaluationRecord; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationRecordSortBy; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationValueType; -import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType; import org.apache.skywalking.oap.server.core.query.enumeration.Order; import org.apache.skywalking.oap.server.core.query.input.Duration; import org.apache.skywalking.oap.server.core.query.input.GenAITraceScopeCondition; @@ -107,19 +106,28 @@ public GenAIEvaluationRecords queryGenAIEvaluationRecord(String serviceId, Strin if (Objects.nonNull(relatedTrace)) { if (Objects.nonNull(relatedTrace.getType())) { where.eq(GenAIEvaluationRecord.REF_TYPE, relatedTrace.getType().name()); + switch (relatedTrace.getType()) { + case SKYWALKING_NATIVE: + if (StringUtil.isNotEmpty(relatedTrace.getSegmentId())) { + where.eq(GenAIEvaluationRecord.SEGMENT_ID, relatedTrace.getSegmentId()); + } + if (Objects.nonNull(relatedTrace.getSpanIndex())) { + where.eq(GenAIEvaluationRecord.SPAN_INDEX, (long) relatedTrace.getSpanIndex()); + } + break; + case OTLP: + case ZIPKIN: + if (StringUtil.isNotEmpty(relatedTrace.getSpanId())) { + where.eq(GenAIEvaluationRecord.SPAN_ID, relatedTrace.getSpanId()); + } + break; + default: + throw new IllegalArgumentException("Unsupported relatedTrace.type: " + relatedTrace.getType()); + } } if (StringUtil.isNotEmpty(relatedTrace.getTraceId())) { where.eq(GenAIEvaluationRecord.TRACE_ID, relatedTrace.getTraceId()); } - if (StringUtil.isNotEmpty(relatedTrace.getSegmentId())) { - where.eq(GenAIEvaluationRecord.SEGMENT_ID, relatedTrace.getSegmentId()); - } - if (GenAITraceRefType.OTLP.equals(relatedTrace.getType()) - && StringUtil.isNotEmpty(relatedTrace.getSpanId())) { - where.eq(GenAIEvaluationRecord.SPAN_ID, relatedTrace.getSpanId()); - } else if (Objects.nonNull(relatedTrace.getSpanIndex())) { - where.eq(GenAIEvaluationRecord.SPAN_INDEX, (long) relatedTrace.getSpanIndex()); - } } if (GenAIEvaluationRecordSortBy.SCORE_VALUE.equals(sortBy)) { diff --git a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java index 34b54b1b4a3a..eea0e11c327c 100644 --- a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java +++ b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/GenAIEvaluationRecordQueryEsDAO.java @@ -30,7 +30,6 @@ import org.apache.skywalking.oap.server.core.analysis.record.Record; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationRecordSortBy; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationValueType; -import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType; import org.apache.skywalking.oap.server.core.query.enumeration.Order; import org.apache.skywalking.oap.server.core.query.input.Duration; import org.apache.skywalking.oap.server.core.query.input.GenAITraceScopeCondition; @@ -112,19 +111,28 @@ public GenAIEvaluationRecords queryGenAIEvaluationRecord(final String serviceId, if (nonNull(relatedTrace)) { if (nonNull(relatedTrace.getType())) { query.must(Query.term(GenAIEvaluationRecord.REF_TYPE, relatedTrace.getType().name())); + switch (relatedTrace.getType()) { + case SKYWALKING_NATIVE: + if (isNotEmpty(relatedTrace.getSegmentId())) { + query.must(Query.term(GenAIEvaluationRecord.SEGMENT_ID, relatedTrace.getSegmentId())); + } + if (nonNull(relatedTrace.getSpanIndex())) { + query.must(Query.term(GenAIEvaluationRecord.SPAN_INDEX, relatedTrace.getSpanIndex())); + } + break; + case OTLP: + case ZIPKIN: + if (isNotEmpty(relatedTrace.getSpanId())) { + query.must(Query.term(GenAIEvaluationRecord.SPAN_ID, relatedTrace.getSpanId())); + } + break; + default: + throw new IllegalArgumentException("Unsupported relatedTrace.type: " + relatedTrace.getType()); + } } if (isNotEmpty(relatedTrace.getTraceId())) { query.must(Query.term(GenAIEvaluationRecord.TRACE_ID, relatedTrace.getTraceId())); } - if (isNotEmpty(relatedTrace.getSegmentId())) { - query.must(Query.term(GenAIEvaluationRecord.SEGMENT_ID, relatedTrace.getSegmentId())); - } - if (GenAITraceRefType.OTLP.equals(relatedTrace.getType()) - && isNotEmpty(relatedTrace.getSpanId())) { - query.must(Query.term(GenAIEvaluationRecord.SPAN_ID, relatedTrace.getSpanId())); - } else if (nonNull(relatedTrace.getSpanIndex())) { - query.must(Query.term(GenAIEvaluationRecord.SPAN_INDEX, relatedTrace.getSpanIndex())); - } } final SearchBuilder search = Search.builder() .query(query) diff --git a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java index 641214b23318..9201f9bb8b17 100644 --- a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java +++ b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCGenAIEvaluationRecordQueryDAO.java @@ -28,7 +28,6 @@ import org.apache.skywalking.oap.server.core.analysis.manual.genai.GenAIEvaluationRecord; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationRecordSortBy; import org.apache.skywalking.oap.server.core.query.enumeration.GenAIEvaluationValueType; -import org.apache.skywalking.oap.server.core.query.enumeration.GenAITraceRefType; import org.apache.skywalking.oap.server.core.query.enumeration.Order; import org.apache.skywalking.oap.server.core.query.input.Duration; import org.apache.skywalking.oap.server.core.query.input.GenAITraceScopeCondition; @@ -242,23 +241,32 @@ protected SQLAndParameters buildSQL(final String serviceId, if (nonNull(relatedTrace.getType())) { sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.REF_TYPE)).append(" = ?"); parameters.add(relatedTrace.getType().name()); + switch (relatedTrace.getType()) { + case SKYWALKING_NATIVE: + if (StringUtil.isNotEmpty(relatedTrace.getSegmentId())) { + sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.SEGMENT_ID)).append(" = ?"); + parameters.add(relatedTrace.getSegmentId()); + } + if (nonNull(relatedTrace.getSpanIndex())) { + sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.SPAN_INDEX)).append(" = ?"); + parameters.add(relatedTrace.getSpanIndex()); + } + break; + case OTLP: + case ZIPKIN: + if (StringUtil.isNotEmpty(relatedTrace.getSpanId())) { + sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.SPAN_ID)).append(" = ?"); + parameters.add(relatedTrace.getSpanId()); + } + break; + default: + throw new IllegalArgumentException("Unsupported relatedTrace.type: " + relatedTrace.getType()); + } } if (StringUtil.isNotEmpty(relatedTrace.getTraceId())) { sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.TRACE_ID)).append(" = ?"); parameters.add(relatedTrace.getTraceId()); } - if (StringUtil.isNotEmpty(relatedTrace.getSegmentId())) { - sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.SEGMENT_ID)).append(" = ?"); - parameters.add(relatedTrace.getSegmentId()); - } - if (GenAITraceRefType.OTLP.equals(relatedTrace.getType()) - && StringUtil.isNotEmpty(relatedTrace.getSpanId())) { - sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.SPAN_ID)).append(" = ?"); - parameters.add(relatedTrace.getSpanId()); - } else if (nonNull(relatedTrace.getSpanIndex())) { - sql.append(" and ").append(storageColumn(GenAIEvaluationRecord.SPAN_INDEX)).append(" = ?"); - parameters.add(relatedTrace.getSpanIndex()); - } } final boolean sortByScore = GenAIEvaluationRecordSortBy.SCORE_VALUE.equals(sortBy); final String sortColumn = storageColumn( diff --git a/test/e2e-v2/cases/mixed-virtual-genai/check-trace-ref.sh b/test/e2e-v2/cases/mixed-virtual-genai/check-trace-ref.sh new file mode 100755 index 000000000000..fd88ebf6c5b3 --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/check-trace-ref.sh @@ -0,0 +1,143 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Usage: check-trace-ref.sh +# +# Picks the newest Faithfulness record of the service, queries it again through relatedTrace, and prints +# which trace stores hold the referenced trace: the SkyWalking trace query, the TraceQL OTLP datasource and +# the Zipkin query API. Only the store the type names should hold it. A store that cannot be queried prints +# error: instead of true or false, so the check fails and is retried. + +set -u + +TYPE=$1 +SERVICE_ID="$(printf '%s' "$2" | base64).1" +GRAPHQL="http://${oap_host}:${oap_12800}/graphql" +START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') +END=$(date -u '+%Y-%m-%d %H%M') +RECORD_QUERY='query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type traceId segmentId spanIndex spanId } serviceName } } }' + +# $1: extra condition fields, starting with a comma. +records() { + curl -s --max-time 10 -X POST "$GRAPHQL" -H 'Content-Type: application/json' \ + -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","serviceId":"'"$SERVICE_ID"'","taskName":"Faithfulness","valueType":"SCORE"'"$1"',"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"EVALUATION_TIME","queryOrder":"DES"}}}' +} + +# $1: JSON document, $2: yq expression on it. +json() { + printf '%s' "$1" | yq -p json "$2" +} + +# $1: URL, $2: the statuses meaning the store does not hold the trace. Prints the body of a 200 response, +# nothing for those statuses, and error: for anything else, including a request curl did not complete. +fetch() { + local body status rc + body=$(mktemp) + status=$(curl -s --max-time 10 -o "$body" -w '%{http_code}' -H 'Accept: application/json' "$1") + rc=$? + if [ "$rc" -ne 0 ]; then + printf 'error:curl-%s' "$rc" + elif [ "$status" = 200 ]; then + cat "$body" + elif ! printf ' %s ' "$2" | grep -q " $status "; then + printf 'error:%s' "$status" + fi + rm -f "$body" +} + +# $1: input format, $2: fetched document, $3: yq expression. A missing document means the store does not hold +# the trace. +holds() { + case "$2" in + error:*) printf '%s' "$2" ;; + "") printf 'false' ;; + *) printf '%s' "$2" | yq -p "$1" "$3" ;; + esac +} + +RECORD=$(json "$(records '')" '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0] // {}' | yq -o json -I 0) +REF_TYPE=$(json "$RECORD" '.traceRef.type // ""') +SERVICE_NAME=$(json "$RECORD" '.serviceName // ""') +TRACE_ID=$(json "$RECORD" '.traceRef.traceId // ""') +SEGMENT_ID=$(json "$RECORD" '.traceRef.segmentId // ""') +SPAN_INDEX=$(json "$RECORD" '.traceRef.spanIndex // ""') +SPAN_ID=$(json "$RECORD" '.traceRef.spanId // ""') + +# The span coordinates of the type, and the same reference pointing at a span that does not exist. +case "$TYPE" in + SKYWALKING_NATIVE) + RELATED='"traceId":"'"$TRACE_ID"'","segmentId":"'"$SEGMENT_ID"'","spanIndex":'"${SPAN_INDEX:-null}" + MISMATCHED='"traceId":"'"$TRACE_ID"'","segmentId":"'"$SEGMENT_ID"'","spanIndex":99999' + SAME_SPAN='.traceRef.traceId == "'"$TRACE_ID"'" and .traceRef.segmentId == "'"$SEGMENT_ID"'" and .traceRef.spanIndex == '"${SPAN_INDEX:--1}" + ;; + OTLP|ZIPKIN) + RELATED='"traceId":"'"$TRACE_ID"'","spanId":"'"$SPAN_ID"'"' + MISMATCHED='"traceId":"'"$TRACE_ID"'","spanId":"0000000000000000"' + SAME_SPAN='.traceRef.traceId == "'"$TRACE_ID"'" and .traceRef.spanId == "'"$SPAN_ID"'"' + ;; + *) + echo "unsupported trace ref type: $TYPE" >&2 + exit 1 + ;; +esac +LOOKUP=$(json "$(records ',"relatedTrace":{"type":"'"$TYPE"'",'"$RELATED"'}')" \ + '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0] // {}' | yq -o json -I 0) +# A failed query must not read as zero records. +MISMATCHED_RESPONSE=$(records ',"relatedTrace":{"type":"'"$TYPE"'",'"$MISMATCHED"'}') +MISMATCHED_LIST='.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList' +if [ "$(json "$MISMATCHED_RESPONSE" '.errors == null and ('"$MISMATCHED_LIST"' | tag) == "!!seq"')" = true ]; then + MISMATCHED_COUNT=$(json "$MISMATCHED_RESPONSE" "$MISMATCHED_LIST | length") +else + MISMATCHED_COUNT="error:graphql" +fi + +SKYWALKING_TRACE="error:no-trace-id" +OTLP_TRACE="error:no-trace-id" +ZIPKIN_TRACE="error:no-trace-id" +if [ -n "$TRACE_ID" ]; then + if ! SKYWALKING_TRACE=$(swctl --display yaml --base-url="$GRAPHQL" trace "$TRACE_ID"); then + SKYWALKING_TRACE="error:swctl" + fi + OTLP_TRACE=$(fetch "http://${oap_host}:${oap_3200}/otlp/api/v2/traces/${TRACE_ID}" "404") + # Zipkin answers 400 for a trace ID that is not lower-hex, such as a SkyWalking native one. + ZIPKIN_TRACE=$(fetch "http://${oap_host}:${oap_9412}/zipkin/api/v2/trace/${TRACE_ID}" "404 400") +fi + +case "$TYPE" in + SKYWALKING_NATIVE) + SPAN_IN_STORE=$(holds yaml "$SKYWALKING_TRACE" \ + '[.spans[] | select(.segmentid == "'"$SEGMENT_ID"'" and .spanid == '"${SPAN_INDEX:--1}"')] | length > 0') + ;; + OTLP) + SPAN_IN_STORE=$(holds json "$OTLP_TRACE" \ + '[.trace.resourceSpans[].scopeSpans[].spans[] | select(.spanId == "'"$SPAN_ID"'")] | length > 0') + ;; + ZIPKIN) + SPAN_IN_STORE=$(holds json "$ZIPKIN_TRACE" '[.[] | select(.id == "'"$SPAN_ID"'")] | length > 0') + ;; +esac + +cat < 0') +traceInOtlp: $(holds json "$OTLP_TRACE" '[.trace.resourceSpans[].scopeSpans[].spans[]] | length > 0') +traceInZipkin: $(holds json "$ZIPKIN_TRACE" 'length > 0') +EOF diff --git a/test/e2e-v2/cases/mixed-virtual-genai/docker-compose.yml b/test/e2e-v2/cases/mixed-virtual-genai/docker-compose.yml new file mode 100644 index 000000000000..88500f81dc42 --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/docker-compose.yml @@ -0,0 +1,144 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# One OAP judging GenAI spans from all three trace stores at once: SkyWalking native segments from the +# Spring AI service, OTLP spans stored natively from provider-otlp, and Zipkin spans from provider-zipkin. +services: + oap: + extends: + file: ../../script/docker-compose/base-compose.yml + service: oap + environment: + SW_OTEL_TRACE_STORAGE: otlp + SW_RECEIVER_ZIPKIN: default + SW_QUERY_ZIPKIN: default + SW_TRACEQL: default + SW_TRACEQL_ENABLE_DATASOURCE_OTLP: "true" + SW_AI_EVALUATION: default + SW_AI_EVALUATION_SAMPLE_RATE: 1000000 + AI_EVALUATION_ENDPOINT: http://mock-llm-server:9090/llm/evaluation/v1/chat/completions + AI_EVALUATION_MODEL: e2e-judge + AI_EVALUATION_API_KEY: e2e-api-key + volumes: + - ../virtual-genai/ai-evaluation.yml:/skywalking/config/ai-evaluation.yml + ports: + - "12800:12800" + - "11800:11800" + - "9411:9411" + - 9412 + - 3200 + networks: + - e2e + + banyandb: + extends: + file: ../../script/docker-compose/base-compose.yml + service: banyandb + ports: + - 17912 + + mock-llm-server: + extends: + file: ../../script/docker-compose/base-compose.yml + service: mock-llm-server + ports: + - 9090 + + spring-ai-service: + image: "ghcr.io/apache/skywalking-java/skywalking-java:${SW_AGENT_JAVA_COMMIT}-java17" + volumes: + - ../../java-test-service/e2e-spring-ai-service/target/e2e-spring-ai-service-2.0.0.jar:/services/app.jar + command: [ "java", "-jar", "/services/app.jar" ] + ports: + - "9260:8080" + networks: + - e2e + environment: + OPENAI_API_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + SPRING_AI_OPENAI_BASE_URL: http://mock-llm-server:9090/llm/v1 + SW_AGENT_COLLECTOR_BACKEND_SERVICES: oap:11800 + SW_LOGGING_OUTPUT: CONSOLE + SW_AGENT_NAME: e2e-spring-ai + SW_AGENT_INSTANCE_NAME: spring-ai-service + SW_PLUGIN_SPRINGAI_COLLECT_INPUT_MESSAGES: "true" + SW_PLUGIN_SPRINGAI_COLLECT_OUTPUT_MESSAGES: "true" + healthcheck: + test: [ "CMD", "sh", "-c", "nc -nz 127.0.0.1 8080" ] + interval: 5s + timeout: 60s + retries: 120 + depends_on: + oap: + condition: service_healthy + mock-llm-server: + condition: service_healthy + + provider-otlp: + build: + context: ../../script/ + dockerfile: dockerfile/Dockerfile-openai.python + networks: + - e2e + ports: + - "9093:9091" + environment: + OPENAI_BASE_URL: http://mock-llm-server:9090/llm/otlp/v1 + OPENAI_API_KEY: xxxxxxxxxxxxxxxxxxxxxxxx + OTEL_SERVICE_NAME: e2e-openai-otlp + OTEL_TRACES_EXPORTER: otlp + OTEL_EXPORTER_OTLP_ENDPOINT: http://oap:11800 + OTEL_EXPORTER_OTLP_PROTOCOL: grpc + depends_on: + oap: + condition: service_healthy + mock-llm-server: + condition: service_healthy + # A GET would make an OpenAI call that the judge evaluates, so the check only connects. + healthcheck: + test: ["CMD", "python3", "-c", "import socket; socket.create_connection(('127.0.0.1', 9091), 2).close()"] + interval: 5s + timeout: 10s + retries: 5 + command: ["opentelemetry-instrument", "python3", "/openai-call.py"] + + provider-zipkin: + build: + context: ../../script/ + dockerfile: dockerfile/Dockerfile-openai.python + networks: + - e2e + ports: + - "9094:9091" + environment: + OPENAI_BASE_URL: http://mock-llm-server:9090/llm/otlp/v1 + OPENAI_API_KEY: xxxxxxxxxxxxxxxxxxxxxxxx + OTEL_SERVICE_NAME: e2e-openai-zipkin + OTEL_TRACES_EXPORTER: zipkin_json + OTEL_EXPORTER_ZIPKIN_ENDPOINT: http://oap:9411/api/v2/spans + depends_on: + oap: + condition: service_healthy + mock-llm-server: + condition: service_healthy + # A GET would make an OpenAI call that the judge evaluates, so the check only connects. + healthcheck: + test: ["CMD", "python3", "-c", "import socket; socket.create_connection(('127.0.0.1', 9091), 2).close()"] + interval: 5s + timeout: 10s + retries: 5 + command: ["opentelemetry-instrument", "python3", "/openai-call.py"] + +networks: + e2e: diff --git a/test/e2e-v2/cases/mixed-virtual-genai/e2e.yaml b/test/e2e-v2/cases/mixed-virtual-genai/e2e.yaml new file mode 100644 index 000000000000..69400b887e78 --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/e2e.yaml @@ -0,0 +1,50 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +setup: + env: compose + file: docker-compose.yml + timeout: 20m + init-system-environment: ../../script/env + steps: + - name: set PATH + command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + - name: install yq + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq + - name: install swctl + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl + # The trigger below drives the SkyWalking-agent service only, so the OTLP and Zipkin providers are called here. + - name: call the OTLP and Zipkin providers + command: | + for i in 1 2 3 4 5; do + curl -s http://localhost:9093/ > /dev/null + curl -s http://localhost:9094/ > /dev/null + sleep 3 + done + +trigger: + action: http + interval: 3s + times: 5 + url: http://localhost:9260/ai/generateStream + method: GET + +verify: + retry: + count: 60 + interval: 3s + cases: + - includes: + - ./trace-ref-cases.yaml diff --git a/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-otlp.yml b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-otlp.yml new file mode 100644 index 000000000000..e723c04bc8bd --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-otlp.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +traceRefType: OTLP +serviceName: e2e-openai-otlp +lookupType: OTLP +lookupSameSpan: true +mismatchedLookupRecords: 0 +spanInItsStore: true +traceInSkyWalking: false +traceInOtlp: true +traceInZipkin: false diff --git a/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-skywalking-native.yml b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-skywalking-native.yml new file mode 100644 index 000000000000..3172b17a169f --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-skywalking-native.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +traceRefType: SKYWALKING_NATIVE +serviceName: e2e-spring-ai +lookupType: SKYWALKING_NATIVE +lookupSameSpan: true +mismatchedLookupRecords: 0 +spanInItsStore: true +traceInSkyWalking: true +traceInOtlp: false +traceInZipkin: false diff --git a/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-types.yml b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-types.yml new file mode 100644 index 000000000000..57b512f00838 --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-types.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- OTLP +- SKYWALKING_NATIVE +- ZIPKIN diff --git a/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-zipkin.yml b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-zipkin.yml new file mode 100644 index 000000000000..42bcde7038d5 --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/expected/trace-ref-zipkin.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +traceRefType: ZIPKIN +serviceName: e2e-openai-zipkin +lookupType: ZIPKIN +lookupSameSpan: true +mismatchedLookupRecords: 0 +spanInItsStore: true +traceInSkyWalking: false +traceInOtlp: false +traceInZipkin: true diff --git a/test/e2e-v2/cases/mixed-virtual-genai/trace-ref-cases.yaml b/test/e2e-v2/cases/mixed-virtual-genai/trace-ref-cases.yaml new file mode 100644 index 000000000000..a21f4c2108ae --- /dev/null +++ b/test/e2e-v2/cases/mixed-virtual-genai/trace-ref-cases.yaml @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cases: + # The judge evaluated spans from all three trace stores of this one OAP. + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql \ + -H 'Content-Type: application/json' \ + -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":50}}}}' \ + | yq -P '[.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[].traceRef.type] | unique | sort' + expected: expected/trace-ref-types.yml + + # For each source, its newest record, the record its relatedTrace finds, and the stores holding its trace. + - query: bash test/e2e-v2/cases/mixed-virtual-genai/check-trace-ref.sh SKYWALKING_NATIVE e2e-spring-ai + expected: expected/trace-ref-skywalking-native.yml + - query: bash test/e2e-v2/cases/mixed-virtual-genai/check-trace-ref.sh OTLP e2e-openai-otlp + expected: expected/trace-ref-otlp.yml + - query: bash test/e2e-v2/cases/mixed-virtual-genai/check-trace-ref.sh ZIPKIN e2e-openai-zipkin + expected: expected/trace-ref-zipkin.yml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml b/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml index 0a14830ffb68..a7aaeaf1c667 100644 --- a/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml +++ b/test/e2e-v2/cases/otlp-virtual-genai/e2e.yaml @@ -42,4 +42,6 @@ verify: cases: - includes: - ./otlp-virtual-genai.yaml + - ./evaluation-storage.yaml + - ./evaluation-span-lookup.yaml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/evaluation-span-lookup.yaml b/test/e2e-v2/cases/otlp-virtual-genai/evaluation-span-lookup.yaml new file mode 100644 index 000000000000..108da2071544 --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/evaluation-span-lookup.yaml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cases: + # Evaluation record span lookup. otlpTraceStorage is zipkin here, so the record + # references the converted span as ZIPKIN. The first query supplies the dynamic + # trace/span reference to the second query so the span_id predicate is exercised + # independently of the broad provider query. + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + RECORD_QUERY='query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type traceId segmentId spanIndex spanId } taskName valueType scoreValue judgeModel } } }' + FIRST=$(curl -s -X POST http://${oap_host}:${oap_12800}/graphql -H 'Content-Type: application/json' -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","minScore":800000,"maxScore":800000,"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"SCORE_VALUE","queryOrder":"DES"}}}') + TRACE_ID=$(printf '%s' "$FIRST" | yq -r '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef.traceId') + SPAN_ID=$(printf '%s' "$FIRST" | yq -r '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef.spanId') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql -H 'Content-Type: application/json' -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","minScore":800000,"maxScore":800000,"relatedTrace":{"type":"ZIPKIN","traceId":"'"$TRACE_ID"'","spanId":"'"$SPAN_ID"'"},"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"SCORE_VALUE","queryOrder":"DES"}}}' | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0] | {"traceType": .traceRef.type, "hasTraceId": (.traceRef.traceId != null and .traceRef.traceId != ""), "hasSpanId": (.traceRef.spanId != null and .traceRef.spanId != ""), "taskName": .taskName, "valueType": .valueType, "scoreValue": .scoreValue, "judgeModel": .judgeModel}' + expected: expected/evaluation-score-record.yml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml b/test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml new file mode 100644 index 000000000000..fad087db29cc --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/evaluation-storage.yaml @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The store holding the judged span, which depends on how this case's OAP receives spans: +# OTLP converted to Zipkin with otlpTraceStorage: zipkin. +cases: + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql \ + -H 'Content-Type: application/json' \ + -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \ + | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef' + expected: expected/evaluation-storage.yml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-score-record.yml b/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-score-record.yml index 8d6b86263106..e097e4959d26 100644 --- a/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-score-record.yml +++ b/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-score-record.yml @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -traceType: OTLP +traceType: ZIPKIN hasTraceId: true hasSpanId: true taskName: Faithfulness diff --git a/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml b/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml new file mode 100644 index 000000000000..97b8729ccff5 --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/expected/evaluation-storage.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: ZIPKIN diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml b/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml index 155957ffaa2a..de766fb41279 100644 --- a/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml +++ b/test/e2e-v2/cases/otlp-virtual-genai/native/e2e.yaml @@ -43,3 +43,5 @@ verify: cases: - includes: - ../otlp-virtual-genai.yaml + - ./evaluation-storage.yaml + - ./evaluation-span-lookup.yaml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-span-lookup.yaml b/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-span-lookup.yaml new file mode 100644 index 000000000000..d1c1ea3c854f --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-span-lookup.yaml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cases: + # Evaluation record span lookup. otlpTraceStorage is otlp here, so the record + # references the natively stored span as OTLP. The first query supplies the dynamic + # trace/span reference to the second query so the span_id predicate is exercised + # independently of the broad provider query. + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + RECORD_QUERY='query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type traceId segmentId spanIndex spanId } taskName valueType scoreValue judgeModel } } }' + FIRST=$(curl -s -X POST http://${oap_host}:${oap_12800}/graphql -H 'Content-Type: application/json' -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","minScore":800000,"maxScore":800000,"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"SCORE_VALUE","queryOrder":"DES"}}}') + TRACE_ID=$(printf '%s' "$FIRST" | yq -r '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef.traceId') + SPAN_ID=$(printf '%s' "$FIRST" | yq -r '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef.spanId') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql -H 'Content-Type: application/json' -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","minScore":800000,"maxScore":800000,"relatedTrace":{"type":"OTLP","traceId":"'"$TRACE_ID"'","spanId":"'"$SPAN_ID"'"},"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"SCORE_VALUE","queryOrder":"DES"}}}' | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0] | {"traceType": .traceRef.type, "hasTraceId": (.traceRef.traceId != null and .traceRef.traceId != ""), "hasSpanId": (.traceRef.spanId != null and .traceRef.spanId != ""), "taskName": .taskName, "valueType": .valueType, "scoreValue": .scoreValue, "judgeModel": .judgeModel}' + expected: expected/evaluation-score-record.yml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml b/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml new file mode 100644 index 000000000000..50dafe80dc2b --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/native/evaluation-storage.yaml @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The store holding the judged span, which depends on how this case's OAP receives spans: +# OTLP stored natively with otlpTraceStorage: otlp. +cases: + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql \ + -H 'Content-Type: application/json' \ + -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \ + | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef' + expected: expected/evaluation-storage.yml diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-score-record.yml b/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-score-record.yml new file mode 100644 index 000000000000..8d6b86263106 --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-score-record.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +traceType: OTLP +hasTraceId: true +hasSpanId: true +taskName: Faithfulness +valueType: SCORE +scoreValue: 800000 +judgeModel: e2e-judge diff --git a/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml b/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml new file mode 100644 index 000000000000..c3322e1ef7fa --- /dev/null +++ b/test/e2e-v2/cases/otlp-virtual-genai/native/expected/evaluation-storage.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: OTLP diff --git a/test/e2e-v2/cases/otlp-virtual-genai/otlp-virtual-genai.yaml b/test/e2e-v2/cases/otlp-virtual-genai/otlp-virtual-genai.yaml index 710523619dc5..a3c1a998dd30 100644 --- a/test/e2e-v2/cases/otlp-virtual-genai/otlp-virtual-genai.yaml +++ b/test/e2e-v2/cases/otlp-virtual-genai/otlp-virtual-genai.yaml @@ -63,16 +63,3 @@ cases: expected: expected/metrics-has-value.yml - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=gen_ai_model_avg_estimated_cost --service-id=b3BlbmFp.0 --instance-name=gpt-4.1-mini-2025-04-14 expected: expected/metrics-has-value.yml - - # OTLP evaluation record span lookup. The first query supplies the dynamic - # trace/span reference to the second query so the OTLP span_id predicate is - # exercised independently of the broad provider query. - - query: | - START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') - END=$(date -u '+%Y-%m-%d %H%M') - RECORD_QUERY='query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type traceId segmentId spanIndex spanId } taskName valueType scoreValue judgeModel } } }' - FIRST=$(curl -s -X POST http://${oap_host}:${oap_12800}/graphql -H 'Content-Type: application/json' -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","minScore":800000,"maxScore":800000,"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"SCORE_VALUE","queryOrder":"DES"}}}') - TRACE_ID=$(printf '%s' "$FIRST" | yq -r '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef.traceId') - SPAN_ID=$(printf '%s' "$FIRST" | yq -r '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef.spanId') - curl -s -X POST http://${oap_host}:${oap_12800}/graphql -H 'Content-Type: application/json' -d '{"query":"'"$RECORD_QUERY"'","variables":{"condition":{"providerId":"b3BlbmFp.0","taskName":"Faithfulness","valueType":"SCORE","minScore":800000,"maxScore":800000,"relatedTrace":{"type":"OTLP","traceId":"'"$TRACE_ID"'","spanId":"'"$SPAN_ID"'"},"queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1},"sortBy":"SCORE_VALUE","queryOrder":"DES"}}}' | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0] | {"traceType": .traceRef.type, "hasTraceId": (.traceRef.traceId != null and .traceRef.traceId != ""), "hasSpanId": (.traceRef.spanId != null and .traceRef.spanId != ""), "taskName": .taskName, "valueType": .valueType, "scoreValue": .scoreValue, "judgeModel": .judgeModel}' - expected: expected/evaluation-score-record.yml diff --git a/test/e2e-v2/cases/virtual-genai/e2e.yaml b/test/e2e-v2/cases/virtual-genai/e2e.yaml index 3416deb91cec..ce6959b2afcf 100644 --- a/test/e2e-v2/cases/virtual-genai/e2e.yaml +++ b/test/e2e-v2/cases/virtual-genai/e2e.yaml @@ -42,4 +42,5 @@ verify: cases: - includes: - ./virtual-genai.yaml + - ./evaluation-storage.yaml diff --git a/test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml b/test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml new file mode 100644 index 000000000000..568af2a5dbf5 --- /dev/null +++ b/test/e2e-v2/cases/virtual-genai/evaluation-storage.yaml @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The store holding the judged span, which depends on how this case's OAP receives spans: +# SkyWalking agent segments. +cases: + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql \ + -H 'Content-Type: application/json' \ + -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \ + | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef' + expected: expected/evaluation-storage.yml diff --git a/test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml b/test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml new file mode 100644 index 000000000000..ff1daf74654d --- /dev/null +++ b/test/e2e-v2/cases/virtual-genai/expected/evaluation-storage.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: SKYWALKING_NATIVE diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml b/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml index 87d47057e20f..b1a3f9acb18c 100644 --- a/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml +++ b/test/e2e-v2/cases/zipkin-virtual-genai/docker-compose.yml @@ -21,6 +21,13 @@ services: environment: SW_QUERY_ZIPKIN: default SW_RECEIVER_ZIPKIN: default + SW_AI_EVALUATION: default + SW_AI_EVALUATION_SAMPLE_RATE: 1000000 + AI_EVALUATION_ENDPOINT: http://mock-llm-server:9090/llm/evaluation/v1/chat/completions + AI_EVALUATION_MODEL: e2e-judge + AI_EVALUATION_API_KEY: e2e-api-key + volumes: + - ../virtual-genai/ai-evaluation.yml:/skywalking/config/ai-evaluation.yml ports: - "12800:12800" - "11800:11800" diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml b/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml index 0147b6d00099..7e9be223fbe4 100644 --- a/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml +++ b/test/e2e-v2/cases/zipkin-virtual-genai/e2e.yaml @@ -42,3 +42,4 @@ verify: cases: - includes: - ./zipkin-virtual-genai.yaml + - ./evaluation-storage.yaml diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml b/test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml new file mode 100644 index 000000000000..09a0871b9f97 --- /dev/null +++ b/test/e2e-v2/cases/zipkin-virtual-genai/evaluation-storage.yaml @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The store holding the judged span, which depends on how this case's OAP receives spans: +# Zipkin spans received by receiver-zipkin. +cases: + - query: | + START=$(date -u -v-1H '+%Y-%m-%d %H%M' 2>/dev/null || date -u -d '1 hour ago' '+%Y-%m-%d %H%M') + END=$(date -u '+%Y-%m-%d %H%M') + curl -s -X POST http://${oap_host}:${oap_12800}/graphql \ + -H 'Content-Type: application/json' \ + -d '{"query":"query ($condition: GenAIEvaluationRecordQueryCondition!) { queryGenAIEvaluationRecord(condition: $condition) { genAIEvaluationRecordList { traceRef { type } } } }","variables":{"condition":{"providerId":"b3BlbmFp.0","queryDuration":{"start":"'"$START"'","end":"'"$END"'","step":"MINUTE"},"paging":{"pageNum":1,"pageSize":1}}}}' \ + | yq -P '.data.queryGenAIEvaluationRecord.genAIEvaluationRecordList[0].traceRef' + expected: expected/evaluation-storage.yml diff --git a/test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml b/test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml new file mode 100644 index 000000000000..97b8729ccff5 --- /dev/null +++ b/test/e2e-v2/cases/zipkin-virtual-genai/expected/evaluation-storage.yml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: ZIPKIN