Skip to content

[Flakiness triage] Run ledger #7

New issue
New issue
Open
Open
[Flakiness triage] Run ledger#7

Description

@mmitche
mmitche
opened on Nov 4, 2026

[Flakiness triage] Run ledger entry

Fixed window: [2026-10-30T17:47:43.872Z, 2026-11-04T17:47:43.872Z)
LOOKBACK_DAYS=5

Population and coverage:

  • 47 merged dotnet/aspnetcore PRs enumerated.
  • 8 unique PRs had 8 distinct final-iteration recovery incidents.
  • 8 recovery builds plus 3 separate failed pipeline-rerun builds were inspected.
  • Initial coverage and one bounded verification pass completed.
  • 5 classified cause groups; 1 incident remained unclassified because its failed timeline returned HTTP 204.

Outcomes:

Draft PRs: none.

Remaining gaps:

  • The collector's deep analyzer rejected tool requests for four cause groups; deterministic retained evidence was used for manual evaluation.
  • The failed timeline for build 1663023 returned HTTP 204 during both evidence passes.

Detailed evidence and run artifacts are preserved in the run's persistent session files.

Activity

  1. mmitche commented on Nov 5, 2026

    @mmitche
    mmitche
    on Nov 5, 2026
    OwnerAuthor

    Fixed window: [2026-10-31T17:02:07.421Z, 2026-11-05T17:02:07.421Z)
    LOOKBACK_DAYS=5

    Population and coverage:

    • 54 merged dotnet/aspnetcore PRs enumerated.
    • 8 unique PRs had 8 distinct final-iteration recovery incidents: 5 stage-attempt recoveries and 3 pipeline reruns.
    • The prior completed checkpoint covered the overlapping population; incremental collection through the fixed end added 7 merged PRs and found no new recovery incidents.
    • Broad initial coverage and one bounded verification pass completed.
    • 5 classified cause groups; 1 incident remains unclassified because failed build 1663023's timeline again returned HTTP 204.

    Outcomes:

    Prototype/draft PR: none. The fork's master remains at 2019 commit f6130e8430d16f614c9ad7e5363e465804c2dd34; upstream main was c6f4231e582f239669b7d06cb3181ac9d0ffdcdf. Importing unrelated upstream history would cost more developer/reviewer time than these candidates can save.

    Remaining gaps:

    • Three retained analyses still fail because tool-less inference unexpectedly requests tools.
    • Blazor test-result errors, the SignalR crash artifact, and build 1663023's failed timeline remain unavailable.

    Detailed evidence and the run summary are preserved in this run's persistent session files.

  2. mmitche commented on Nov 6, 2026

    @mmitche
    mmitche
    on Nov 6, 2026
    OwnerAuthor

    Fork-refresh follow-up: all six existing issues re-evaluated; one fork-only draft opened.

    This implements the user's requested follow-up before another automation run. No CI collection, retry, or automation run was started; no new occurrences are claimed. The original run settings remain LOOKBACK_DAYS=5, fixed window [2026-10-31T17:02:07.421Z, 2026-11-05T17:02:07.421Z). Its recorded population was 54 merged PRs and eight recovery incidents; population/attempt coverage and evidence gaps were not re-measured here.

    • #1: passkey response-body race: blocker removed by the updated fork; draft #8 captures HTML before automatic navigation. Bounded browser red/green evidence and C# API compilation support the small test-only patch; the full generated-application theory remains blocked by the missing repository SDK and matching build artifacts.
    • #2: Kestrel startup race: no competing fork patch. Read the actual fix in dotnet/aspnetcore#67714, which is still open/draft, not merged. Completing that work has better value than maintaining a duplicate.
    • #3: Helix timeline 503: no ASP.NET Core hardening; external service/tool ownership and no demonstrated targeted repository correction.
    • #4: SignalR macOS work-item crash: unresolved; console/dump/stack needed to identify the failing test/process.
    • #5: runtime download timeout: no speculative timeout/retry patch; owning download task already has a retry path, but URI/attempt/service evidence is missing.
    • #6: failed timeline unavailable: unresolved; the initiating failure is still unknown.

    Publication: mmitche/AspNetCore default/base main, fork base 2e792d799058bf25990ce0c8e2b416c736431ca3, candidate commit efb014b0ead8bd729af9e780c3c35f43676aeecc, branch mmitche-ci-flakiness-triage. The diff contains one test file, not a fork catch-up. No upstream writes, issue closures, or draft-readiness changes.

    Detailed review, tradeoffs, reproduction scripts/logs, API probe, and patch are preserved in this run's persistent session files as 2026-11-06-aspnetcore-fix-reevaluation.md and supporting artifacts. This draft is not a claim of verified CI remediation.

  3. mmitche commented on Nov 6, 2026

    @mmitche
    mmitche
    on Nov 6, 2026
    OwnerAuthor

    At the user's explicit request, the reviewed passkey fix is now open as upstream draft dotnet/aspnetcore#70362, targeting upstream main from the same fork branch and commit efb014b. The one-file diff and validation limitations are unchanged. The original fork draft #8 remains open; no issue closure, ready-for-review change, or manual CI retry was performed.

  4. mmitche commented on Nov 6, 2026

    @mmitche
    mmitche
    on Nov 6, 2026
    OwnerAuthor

    At the user's explicit request, fork issue #1 was copied to dotnet/aspnetcore#70363 because GitHub rejected a direct transfer between different owners. The upstream copy preserves the occurrence markers, links to the original discussion, and records the missing CI exception text. It is closed as a duplicate of dotnet/aspnetcore#69494; fork #1 is also closed with a redirect. Neither closure claims a verified fix. Proposed correction: draft dotnet/aspnetcore#70362.

    No new CI occurrences or collection run are claimed by this migration; the prior run's population, window, and coverage gaps remain unchanged. This one-off user-authorized upstream publication does not change the automation's fork-only write boundary. The original issue history and occurrence identifiers remain preserved.

  5. mmitche commented on Nov 6, 2026

    @mmitche
    mmitche
    on Nov 6, 2026
    OwnerAuthor

    Fixed window: [2026-11-01T17:02:39Z, 2026-11-06T17:02:39Z)
    LOOKBACK_DAYS=5

    Population and coverage

    • Enumerated all 72 pull requests merged to dotnet/aspnetcore in the fixed window. Authoritative final head commits and PR target branches came from GitHub GraphQL metadata.
    • Collected final-head GitHub status/check history for all 72 PRs. The 30 non-passing check runs were 25 failures and 5 cancellations; no GitHub-only candidate met the same-iteration flake evidence gate.
    • Matched 214 Azure DevOps builds across 66 PRs to the final PR head using triggerInfo["pr.sourceSha"]. Build discovery had no errors.
    • Retrieved current timelines for all 214 builds and 11 prior-attempt timelines for 10 successful builds. No current- or prior-timeline retrieval gaps remained.
    • Recovery evidence consists of 2 full pipeline reruns and 10 successful builds with failed prior stage attempts. Two successful builds each contained a second independent cause, producing 14 independent recovered cause occurrences across 12 recovery contexts.

    Recovered occurrences

    PR PR target branch Build(s) Notes
    dotnet/aspnetcore#70119 main failed 1656560 / recovery 1656681 Passkey template response-body race; tracked by closed fork #1 and canonical dotnet#69494.
    dotnet/aspnetcore#70108 main failed stage attempt 1 / recovery attempt 2 in 1657221 Passkey template response-body race; fork #1 / dotnet#69494.
    dotnet/aspnetcore#70106 main failed stage attempt 1 / recovery attempt 2 in 1656230 WebApplicationFactory/Kestrel startup race; fork #2; existing draft dotnet#67714.
    dotnet/aspnetcore#70134 main failed 1656818 / recovery 1656925 Helix monitor exited after an Azure DevOps timeline HTTP 503; fork #3.
    dotnet/aspnetcore#70215 release/10.0 failed stage attempt 1 / recovery attempt 2 in 1658663 Runtime artifact download exceeded Arcade's 100-second HTTP timeout; fork #5.
    dotnet/aspnetcore#70263 main failed stage attempt 1 / recovery attempt 2 in 1662505 Passkey template response-body race; fork #1 / dotnet#69494.
    dotnet/aspnetcore#70273 release/10.0 failed stage attempt 1 / recovery attempt 2 in 1662535 SignalR macOS work-item crash without retained stack; fork #4.
    dotnet/aspnetcore#70277 release/11.0 failed stage attempt 1 / recovery attempt 2 in 1662775 Static Web Assets test project failed; exact test/error unavailable; new fork #12.
    dotnet/aspnetcore#70284 release/10.0 failed stage attempt 1 / recovery attempt 2 in 1662960 Five unrelated macOS work items had two-minute dotnet command terminations with exit code 130; new fork #10.
    dotnet/aspnetcore#70282 release/10.0 failed stage attempt 1 / recovery attempt 2 in 1662957 Components endpoint redirect test wrote form HTML instead of an empty body; new fork #9.
    dotnet/aspnetcore#70282 release/10.0 failed stage attempt 1 / recovery attempt 2 in 1662957 Separate macOS CORS discovery work item had the same two-minute exit-code-130 signature as PR 70284; new fork #10.
    dotnet/aspnetcore#69925 main failed stage attempt 1 / recovery attempt 2 in 1661466 IIS recycling test lost a response connection while sending requests during app-pool recycle; new fork #11.
    dotnet/aspnetcore#70322 main failed stage attempt 1 / recovery attempt 2 in 1665105 Verified recurrence of the WebApplicationFactory/Kestrel startup race; occurrence added to fork #2.
    dotnet/aspnetcore#70322 main failed stage attempt 1 / recovery attempt 2 in 1665105 Separate Templates.Blazor.Tests failure; exact test/error unavailable and not attributed to the known passkey race; new fork #13.

    Evaluation and publication

    No source change, prototype, commit, push, or new draft pull request was produced. The fork was verified safely fast-forwardable, but synchronization was unnecessary because no candidate passed the value and confidence gate.

    Reconciliation and remaining gaps

    • The previous ledger entry omitted five successful-build retry contexts now recovered through previousAttempts[].timelineId. Those five builds contained seven independent causes; all are now tracked, including the second independent failures in builds 1662957 and 1665105.
    • Anonymous Azure build, timeline, and task-log APIs were available, but Azure Test Runs and HTML test results redirected to sign-in. That blocks exact failure identification for [Flakiness triage] Static Web Assets test failure details unavailable #12 and [Flakiness triage] Blazor template test failure details unavailable #13.
    • This Windows host cannot reproduce macOS or Linux Helix images, hosted Azure services/agents, the IIS NewShim environment, matching containers/browsers, or provider telemetry. No speculative local validation is reported as CI validation.

    Detailed evidence and the final run summary are preserved in this run's persistent session files.

  6. mmitche commented on Nov 7, 2026

    @mmitche
    mmitche
    on Nov 7, 2026
    OwnerAuthor

    Run: 2026-11-04 through 2026-11-07

    Population window: [2026-11-04T17:01:19.653Z, 2026-11-07T17:01:19.653Z) with LOOKBACK_DAYS=3.

    Coverage

    • Enumerated all 48 PRs merged into dotnet/aspnetcore during the fixed window, using merge time and each PR's authoritative final head and target branch.
    • Matched 138 final-iteration Azure builds across 43 PRs by triggerInfo["pr.sourceSha"]: 72 succeeded, 60 partially succeeded, and 6 failed.
    • Retrieved all 138 current timelines, 27 referenced previous-attempt timeline pairs, all 32 failed retry-task logs, and 15 unique retry Helix producer consoles.
    • Inspected all six final failed builds and correlated the outstanding macOS Helix monitor failure to its producer console.
    • Five PRs had no verifiable Azure association from the final-head metadata:
    PR PR target branch Coverage gap
    dotnet/aspnetcore#69280 main No Azure build could be authoritatively associated with the final head.
    dotnet/aspnetcore#68110 main No Azure build could be authoritatively associated with the final head.
    dotnet/aspnetcore#67088 main No Azure build could be authoritatively associated with the final head.
    dotnet/aspnetcore#69964 main No Azure build could be authoritatively associated with the final head.
    dotnet/aspnetcore#69891 release/8.0 No Azure build could be authoritatively associated with the final head.

    Reconciled flakiness

    The run found 19 independent cause groups. These account for 28 recovered build/cause occurrences. Three additional final failed builds had the already-tracked NavigationLock signature, so they are supporting unrecovered evidence rather than recovered flakes. Downstream monitor failures were not counted independently.

    Cause Tracking Affected build occurrence(s) Decision
    Kestrel/WebApplicationFactory address race #2 1665105, 1668622 Existing upstream fix draft dotnet#67714; annotated the new 1668622 occurrence and did not create competing work.
    macOS Helix commands terminated after two minutes #10 1663994 Grouped nine work items as one host-stall occurrence; deferred pending hosted macOS worker telemetry.
    Templates.Blazor.Tests failure with unavailable retained details #13 1665105 Existing tracking retained; no duplicate report or unsupported diagnosis.
    NavigationLock external navigation remains on current page #14 Recovered: 1658098, 1663164, 1665799, 1666941, 1666987. Unrecovered: 1666467, 1667457, 1667612. Tracks dotnet#66041 and merged re-quarantine dotnet#70369; no competing patch.
    TempData cookie test uses a detached DOM node #15 1660004 Tracks dotnet#70085; no competing patch.
    Node/V8 native crash after Rollup #16 1655692 Deferred pending a native crash dump and exact runtime correlation.
    Shared-framework _AddFilesToNuGetPackage MSBuild cycle #17 1659438 Deferred pending a recurrence with a binary log.
    ResultsCache generated-code test hang #18 1659869 Deferred pending the captured hang dump or another recurrence.
    TempData session test uses a stale element #19 1659871 Tracks dotnet#69996; no competing patch.
    Variable-height table virtualization misses rendered rows #20 1659871, 1663164, 1665799, 1666987 main fix dotnet#70243 already exists; all observed PRs targeted release/11.0, so a fork-main prototype would not address them.
    NavigationLock continue test misses expected element #21 1663164 One recovered occurrence with no bounded cause; deferred pending recurrence or browser trace.
    Virtualization anchor test cannot observe viewport item #22 1663164 One recovered occurrence with no bounded cause; deferred pending recurrence or browser trace.
    CoreCLR clsload.cpp assertion #23 1666465 Runtime/toolchain-owned unless a retained dump proves an ASP.NET trigger.
    npm/Azure Artifacts ECONNRESET #24 1663994, 1663995 External feed/network interruption; no extra repository retry layer.
    Shared-framework GetFilesToPublish MSBuild cycle #25 1663994 Kept separate from #17 because the target and branch differ and no binlog proves shared identity; deferred pending a binlog.
    HTTP/3 shutdown test misses expected control-stream log #26 1667664 Deferred pending recurrence or QUIC trace.
    Windows NuGet assets-file collision #27 1667789 Tracks dotnet#61178; no competing patch.
    Windows certificate export fails during reload test #28 1667789 Deferred pending hosted Windows certificate/key-state evidence.
    Gradle distribution DNS failure #29 1665741 External DNS outage after the wrapper's existing retries; no repository change.

    Final failed builds

    A final failed build is not automatically a flake. All six were inspected:

    PR PR target branch Build(s) Disposition
    dotnet/aspnetcore#69827 main 1661294 QuickGrid_AnchorMode_NearTop_PrependKeepsViewportStable failed its scroll-top assertion. No same-iteration recovery or exact matching cause was established.
    dotnet/aspnetcore#69827 main 1661292 macOS Arm64 test host exited unexpectedly while multiple analyzer tests were active. The console has no causal stack or retained dump, so the failing test is unknown.
    dotnet/aspnetcore#70294 main 1666467 Both attempts failed with the NavigationLock signature tracked in #14.
    dotnet/aspnetcore#70344 main 1667457, 1667612 Both final builds failed with the NavigationLock signature tracked in #14.
    dotnet/aspnetcore#70208 release/8.0 1658468 Debian installer build failed creating the NuGet-Migrations mutex with System.IO.IOException. No same-iteration recovery or causal repository defect was established.

    Prototype and validation decision

    No prototype or draft PR passed the value and confidence gate. Existing upstream fixes or tracking already cover the bounded cases; the remaining cases are external service/toolchain failures or lack the dumps, binlogs, browser traces, and platform evidence needed for a safe correction. The fork base was therefore not synchronized merely for maintenance, and no source files, commits, branches, pushes, or pull requests were created.

    Host-specific validation gaps include the hosted macOS and Linux Helix workers, Azure pipeline services and package feeds, matching browser/server matrices, QUIC infrastructure, the relevant containers, runtime and hang dumps, MSBuild binary logs, and hosted Windows certificate/profile state. No local test run could reproduce these CI-only conditions.

    All GitHub writes were limited to mmitche/AspNetCore; upstream remained read-only.

  7. mmitche commented on Nov 8, 2026

    @mmitche
    mmitche
    on Nov 8, 2026
    OwnerAuthor

    ASP.NET Core last-iteration CI flakiness triage

    Population window: [2026-11-05T17:01:42.322Z, 2026-11-08T17:01:42.322Z) with LOOKBACK_DAYS=3.

    Coverage

    • Enumerated all 44 pull requests merged into dotnet/aspnetcore during the fixed window.
    • Retrieved every page of final-head GitHub check history; 22 PRs required a second 100-record page.
    • Authoritatively matched 141 Azure DevOps builds across 40 PRs using both the GitHub final head and Azure triggerInfo["pr.sourceSha"] / triggerInfo["pr.number"].
    • Build outcomes: 67 succeeded, 58 partially succeeded, 12 canceled, and 4 failed.
    • Retrieved all 141 current timelines, all 26 referenced previous-attempt timelines for 20 retry contexts, and all 30 failed retry-task logs.
    • Sixteen historical Azure build IDs returned HTTP 404. They affect four PRs and are retained as coverage gaps rather than treated as passing evidence.
    PR PR target branch Coverage gap
    dotnet/aspnetcore#68110 main Builds 1617532 and 1617533 are no longer available from the Azure API.
    dotnet/aspnetcore#67088 main Builds 1597375-1597378 and 1625584-1625587 are no longer available from the Azure API.
    dotnet/aspnetcore#69964 main Builds 1644491-1644493 are no longer available from the Azure API.
    dotnet/aspnetcore#69891 release/8.0 Builds 1638521, 1638524, and 1638526 are no longer available from the Azure API.

    Reconciliation

    Nineteen builds contained recovered retry contexts. They represented 26 independent recovered cause occurrences. One additional retry context, build 1666467, failed both attempts and is supporting evidence for the existing NavigationLock issue rather than a recovered flake.

    All overlapping occurrences were already reconciled by the prior run to fork issues #2, #10, and #13-#29. Two newly admitted PRs added new occurrences to existing issues:

    PR PR target branch Build(s) Notes
    dotnet/aspnetcore#70163 main Failed attempt 1 and recovery attempt 2 in 1658023 NavigationLock external navigation remained at /subdir/; annotated fork #14.
    dotnet/aspnetcore#70182 main Failed attempt 1 and recovery attempt 2 in 1660106 Variable-height table test found no rendered rows; annotated fork #20.

    The four final failed builds were also inspected. Builds 1666467, 1667457, and 1667612 have the already-tracked NavigationLock signature. Build 1658468 failed creating the NuGet-Migrations mutex, but no same-iteration recovery or causal repository defect was established.

    Evaluation, prototype, and review

    No candidate offered a credible net developer-time benefit over existing fixes, tracking, or external-owner follow-up. Therefore no prototype, source edit, commit, push, fork synchronization, or draft PR was produced.

    Validation and infrastructure gaps

    Evidence validation used GitHub final-head metadata and public Azure build, timeline, and task-log APIs. This Windows host cannot reproduce hosted macOS or Linux Helix workers, Azure pipeline services and package feeds, matching browser/server matrices, QUIC infrastructure, relevant containers, runtime or hang dumps, MSBuild binary logs, or hosted Windows certificate/profile state. No local run is represented as CI remediation validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions