Skip to content

FlightCheck: classify ADK invocations as DA - #470

Merged
amilandi merged 6 commits into
mainfrom
amilandin/flightcheck-adk-da-attribution
Oct 30, 2026
Merged

amilandi merged 6 commits into
mainfrom
amilandin/flightcheck-adk-da-attribution

Conversation

@amilandi

@amilandi amilandi commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • classify every FlightCheck run invoked with --invocation-source adk as declarative_agent
  • preserve custom_agent attribution for runs from main-ca
  • emit telemetry from the early native-DA maker-diagnostics path, which previously returned before both telemetry families
  • map diagnostic outcomes to the existing ADK pass / partial / fail taxonomy
  • keep existing branch-based attribution for CLI, installer, and connect runs

Root cause

ADK /flightcheck runs had two independent gaps:

  1. Runs from topic branches emitted agentType=unknown, so DA dashboards filtered them out.
  2. Native DA diagnostics exited before the legacy FlightCheck and additive ADK telemetry blocks, so no event was sent at all.

The ADK invocation path is authoritative for DA attribution; only the CA shipping branch overrides it.

Live debugging

An instrumented hybrid/local run posted 52 PROD events and received HTTP 204, confirming the collector path is healthy. Raw FlightCheck Kusto remains delayed, with its newest visible row currently dated October 26.

Validation

  • Manually triggered a flightcheck from this branch and validated that it correctly propagates to aria
  • python -m pytest -q tests/flightcheck/test_telemetry.py tests/flightcheck/test_cli.py (154 passed)
  • python -m ruff check solutions/ess-maker-skills/scripts/flightcheck/telemetry.py solutions/ess-maker-skills/scripts/flightcheck/cli.py tests/flightcheck/test_telemetry.py tests/flightcheck/test_cli.py
  • git diff --check

Avery Milandin added 3 commits October 28, 2026 09:34
Treat FlightCheck runs invoked through the ADK slash-command path as declarative-agent telemetry on every branch except main-ca. Preserve branch-based attribution for CLI, installer, and connect runs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 95071db8-bd9f-4036-b163-165d5a3dcf09
Send legacy FlightCheck and additive ADK events for the early DA maker-diagnostics path. Preserve diagnostic-only reporting while mapping its bounded outcomes to pass, partial, or fail telemetry.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 95071db8-bd9f-4036-b163-165d5a3dcf09
Emit telemetry for selected DA checkpoints and profiles, retain verified tenant and selected-agent identity, and align connector and invocation-aware agent-type attribution across both telemetry families.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a4cd1deb-cafc-4d91-ab4a-5c171fcd1108

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Request changes: the diagnostic outcome mapping is sound, but the new telemetry path can misattribute the selected agent, tenant, and connector, and the invocation-aware agent classification reaches only the legacy event family. Static review of f6dd640; tests and live dashboard behavior were not independently verified.

Comment thread solutions/ess-maker-skills/scripts/flightcheck/cli.py
Comment thread solutions/ess-maker-skills/scripts/flightcheck/telemetry.py
Comment thread solutions/ess-maker-skills/scripts/flightcheck/cli.py Outdated
Comment thread solutions/ess-maker-skills/scripts/flightcheck/cli.py Outdated

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Request changes at 7a3d6aae7160f3e9fd9c75ffa89ca1e1853a1204.
The shared invocation-aware classification is sound, and all four previous attribution findings are fixed for their original triggers.
I withdraw those findings.
One P2 integration issue remains: newly emitted native diagnostics can be reported as Ready by the legacy projection, and the checked-in readiness queries misclassify diagnostic outcomes.
Details and the suggested fix are in the inline comment.
Static review only; tests, builds, live telemetry, and deployed dashboards were not independently exercised.

Comment thread solutions/ess-maker-skills/scripts/flightcheck/cli.py
Preserve inspection-specific legacy outcome labels, including skipped-only local diagnostics. Restrict time-to-green and tenant ready-rate cohorts to explicit readiness verdicts. Cover real DA local producers through both telemetry payloads and guard query predicates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a4cd1deb-cafc-4d91-ab4a-5c171fcd1108

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Approve at 53285b75c0fef722bf0b36e8e3a5756621e8a539. No high-risk blocker identified. Previous attribution findings and the original diagnostic-outcome defects are fixed. The remaining connector success-rate issue is non-blocking and recorded in the existing thread for a separate follow-up PR. Static review only; tests, live telemetry, and deployed dashboards were not independently verified.

Resolve the telemetry guide conflict by retaining diagnostic outcome documentation alongside the newly merged session-recording and validation guidance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a4cd1deb-cafc-4d91-ab4a-5c171fcd1108
@amilandi
amilandi merged commit aaf58cf into main Oct 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants