Summary
The search bar evaluates AND at the span/row level. There is no way to express a trace-level filter: return traces where property A appears on some span and property B appears on a (possibly different) span in the same trace.
That is a common query when correlation IDs are stamped once at the edge and service name (or other attributes) live on downstream spans.
Problem
A query like:
SpanAttributes.tuid:"abc123" AND ServiceName:checkout-service
Expected (Jaeger / Haystack / many APM UIs): traces that contain both predicates, even if they sit on different spans.
Actual: 0 results unless a single span has both fields.
This is easy to misread as a bug. The search bar is doing what it was built to do — filtering rows. The gap is that trace-level, cross-span AND is not expressible there.
Why this happens
Typical OpenTelemetry instrumentation stamps a correlation ID (request ID, tenant ID, user ID, etc.) on the gateway/entry span. Downstream services appear as separate spans in the same trace and often do not copy that ID onto every child.
So a real trace can look like:
| Span |
ServiceName |
tuid |
| entry |
gateway-service |
abc123 |
| child |
checkout-service |
(unset) |
- Trace-level
AND matches this trace
- Span-level
AND matches nothing
Current workaround
Intersect two TraceId sets outside the where bar:
TraceIds where any span has tuid = XTraceIds where any span has ServiceName = Y- Fetch full traces for the intersection
That works against ClickHouse, but it is not something a user can do from the search bar.
Proposed behavior
Support a trace-scoped search mode (or syntax) where boolean operators apply at the trace:
A AND B → traces that contain a span matching A and a span matching B- Keep today’s span-level
AND as an explicit mode, since it is still useful (“this span is slow and this span is checkout-service”)
Jaeger, Haystack, and several commercial APM products default to the trace-level interpretation for this class of query.
Additional context
Verified directly in ClickHouse on otel_traces-style data: the traces exist; no single row satisfies both predicates. Happy to share a minimal SQL sketch of the TraceId intersection if useful.
Summary
The search bar evaluates
ANDat the span/row level. There is no way to express a trace-level filter: return traces where property A appears on some span and property B appears on a (possibly different) span in the same trace.That is a common query when correlation IDs are stamped once at the edge and service name (or other attributes) live on downstream spans.
Problem
A query like:
Expected (Jaeger / Haystack / many APM UIs): traces that contain both predicates, even if they sit on different spans.
Actual: 0 results unless a single span has both fields.
This is easy to misread as a bug. The search bar is doing what it was built to do — filtering rows. The gap is that trace-level, cross-span
ANDis not expressible there.Why this happens
Typical OpenTelemetry instrumentation stamps a correlation ID (request ID, tenant ID, user ID, etc.) on the gateway/entry span. Downstream services appear as separate spans in the same trace and often do not copy that ID onto every child.
So a real trace can look like:
ANDmatches this traceANDmatches nothingCurrent workaround
Intersect two
TraceIdsets outside the where bar:TraceIds where any span hastuid = XTraceIds where any span hasServiceName = YThat works against ClickHouse, but it is not something a user can do from the search bar.
Proposed behavior
Support a trace-scoped search mode (or syntax) where boolean operators apply at the trace:
A AND B→ traces that contain a span matching A and a span matching BANDas an explicit mode, since it is still useful (“this span is slow and this span ischeckout-service”)Jaeger, Haystack, and several commercial APM products default to the trace-level interpretation for this class of query.
Additional context
Verified directly in ClickHouse on
otel_traces-style data: the traces exist; no single row satisfies both predicates. Happy to share a minimal SQL sketch of theTraceIdintersection if useful.