Skip to content

Fix missing session starts in capability-first ADK telemetry - #449

Merged
amilandi merged 8 commits into
mainfrom
amilandin/fix-capability-session-telemetry
Oct 29, 2026
Merged

amilandi merged 8 commits into
mainfrom
amilandin/fix-capability-session-telemetry

Conversation

@amilandi

@amilandi amilandi commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Capability-first ADK entry points allocate and persist a session ID without emitting adk.session.start. A later explicit start_session() previously treated the allocated ID as evidence that a start had already been emitted and skipped it. Capability usage could therefore appear in External DA charts while sessions and active installs remained zero.

Changes

  • Separate session-ID allocation from persisted start-event recording. Ensure starts in shared single-event and validated client-batch dispatch, rather than relying on individual skill wrappers.
  • Match starts to the activity's install, tenant, agent type, and telemetry project. An earlier anonymous, CEA, other-install, or dev-project start cannot suppress the start required by another dashboard cohort.
  • Mark a context recorded only after delivery or successful buffering; retry starts that could neither be sent nor buffered. Preserve opt-out, async dispatch, the 30-minute inactivity window, and existing event dimensions.
  • Reserve session starts under short local-state locks, buffer follower activity while a start is pending, and expire abandoned reservations. Atomically detach retry-buffer snapshots before collector I/O so slow flushes do not block or lose concurrent appends. Live events join the queue while an older snapshot is in flight, preserving start-before-activity ordering.
  • Document the reporting semantics and best-effort delivery limits. Active-user charts count distinct installs, not identified people. A session spanning attribution contexts may have a start in each; deduplicate session IDs when combining cohorts.

Verification

381 targeted tests passed across the SDK, authentication, Workday telemetry, FlightCheck telemetry, and telemetry guard suites. Regression coverage includes capability-first starts, matching External DA attribution, legacy/allocated session files, timeout rollover, consent changes, buffered replay, failed buffering, async responsiveness, cohort changes, and simultaneous fresh capability-shim subprocesses. Ruff and diff whitespace checks pass.

Merge order

PR #458 is merged, and this branch is rebased onto it with both tenant-classification and session-reporting documentation/tests preserved. Rebase #436 after #449 so its LPC emitter changes are applied to this finalized transport.

Scope

Prospective emitter fix only: existing clones must pull the change; historical missing session-start events are not backfilled. No Aria cube or FlightCheck outcome-event schema changes. Dashboard test-tenant exclusions were saved separately and are not repository changes.

@amilandi
amilandi marked this pull request as ready for review October 26, 2026 22:01
@nkemms

nkemms commented Oct 26, 2026

Copy link
Copy Markdown
Contributor
  • This change separates session allocation from start recording and adds useful regression coverage, but the new cross-process locks are held across collector network calls.
  • Do not merge: 0 CRITICAL and 3 MAJOR findings. Two reproduced lock-timeout paths violate ordering or lose telemetry; active PRs 458 and 436 also have textual merge conflicts with this branch.
  • Targeted validation passed: 301 tests, Ruff, and diff whitespace checks.

Verdict: Do not merge until network I/O is moved outside the session-start and buffer file locks, contention tests cover waits beyond the lock deadline, and merge order is coordinated with PRs 458 and 436.

[MAJOR] F-3 — Active telemetry PRs have textual merge conflicts
Owner: Dev | Confidence: High | Impact: Medium

git merge-tree reports conflicts with PR 458 in TELEMETRY.md and tests/test_adk_telemetry.py, and with PR 436 in adk_telemetry.py. Both siblings are open and non-draft. PR 458 is current and shares the same author; PR 436 adds LPC operation telemetry to the same emitter.

Fix: Choose a merge order now. Rebase whichever PR merges second and preserve both telemetry changes and their tests.

Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py
@amilandi

Copy link
Copy Markdown
Contributor Author

Addressed the concurrency findings in 2412c309 and selected the merge order for F-3: #458 → #449 → #436. PR #449 will be rebased after #458 merges, preserving both TELEMETRY.md/test changes; #436 should then rebase onto the finalized transport.

The new contention tests hold collector calls for 6.2 seconds, exceeding the former five-second lock deadline. Validation: 368 passed; Ruff and whitespace checks clean.

Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py Outdated
Avery Milandin and others added 2 commits October 27, 2026 10:21
Track recorded session starts separately from allocated IDs and ensure starts from shared single-event and batch dispatch. Preserve dashboard attribution across install, tenant, agent-type, and collector contexts; coordinate concurrent CLI processes and retain retries when buffering fails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
Reserve session starts under short locks, buffer followers while starts are pending, and detach retry snapshots before collector I/O so concurrent appends cannot be lost or overtaken.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 95071db8-bd9f-4036-b163-165d5a3dcf09
@amilandi
amilandi force-pushed the amilandin/fix-capability-session-telemetry branch from 2412c30 to 2ef0a9c Compare October 27, 2026 17:22
@amilandi

Copy link
Copy Markdown
Contributor Author

Resolved the merge conflicts by rebasing onto current main after #458 merged. Preserved both the internal-test-tenant attribution section/tests and the session-start transport section/tests. Rebased head: 2ef0a9c7. Validation: 381 focused tests passed; Ruff and diff whitespace checks are clean.

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Copilot-review] Request changes at 2ef0a9c762de7ad6d7eddb70df15d622d75af364. Separating session allocation from recorded starts and moving network I/O outside state locks are sound improvements. The snapshot-ownership race can still lose a buffered start. The existing normal-exit loss thread remains applicable: #449 (comment). Cross-context replay also violates the documented ordering guarantee. Static review only; no tests, builds or live telemetry were run.

Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py Outdated
Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py Outdated
Batch fresh session starts with owner activity, fence retry snapshots with tokenized claims, and retain session prerequisites through buffered replay.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 95071db8-bd9f-4036-b163-165d5a3dcf09
nkemms
nkemms previously approved these changes Oct 27, 2026

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Approving d3035ad9 with the three unresolved findings below treated as follow-up work. The three earlier blockers are fixed in their demonstrated forms. The session-expiry recovery issue remains unresolved; this approval does not mean it was fixed. Static review only; no tests, builds or live telemetry were run.

Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py
Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py
Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py Outdated
Preserve completed session-start evidence across rollover, isolate prerequisite wrappers from the legacy retry format, and enforce retry-buffer count and byte limits while retaining session starts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 95071db8-bd9f-4036-b163-165d5a3dcf09

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Request changes at 31ef143e8fbdd6516a104e98fd926752a7118662. The shared session-start enforcement is the right design, and the previous rollover-evidence, legacy-buffer-format and buffer-budget findings are fixed in their demonstrated forms. One blocking retry-selection defect remains: eligible buffered activity can be starved indefinitely while later events keep uploading. Completion-ledger growth is a separate nonblocking follow-up. Static review only; no tests, builds or live telemetry were run.

Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py
Comment thread solutions/ess-maker-skills/scripts/adk_telemetry.py
Promote waiting followers after raw retry delivery, quarantine malformed rows, preserve session prerequisites on batch faults, and keep completion failures fail-open. Bound completion evidence with ordered recovery starts for late followers and document deterministic offline validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a4cd1deb-cafc-4d91-ab4a-5c171fcd1108

@daeunJe0ng Dawn Jeong (daeunJe0ng) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[copilot-review] Approve at c5b58f09f87384d46b80e9de8f2fa9f1172421d1.
The previous retry-starvation blocker is fixed: successful raw-snapshot finalization now promotes eligible waiting followers.
Completion history is bounded, and recovery after evidence eviction preserves the original follower attribution and start-before-activity ordering.
I withdraw the previous findings for their demonstrated triggers, including the completion-history follow-up.
The inspected regression tests cover the public-emission retry path and eviction recovery; no remaining Blocking or Should-fix defect was established.
Static review only; tests, builds, and live telemetry were not independently exercised.

@amilandi
amilandi merged commit 309b0b7 into main Oct 29, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants