Skip to content

Don't let one caller's cancellation abort a shared Syft run - #1906

Draft
Suneel Freimuth (suneel-ms) wants to merge 1 commit into
microsoft:mainfrom
suneel-ms:suneel-ms/shared-syft-run-cancellation
Draft

Suneel Freimuth (suneel-ms) wants to merge 1 commit into
microsoft:mainfrom
suneel-ms:suneel-ms/shared-syft-run-cancellation

Conversation

@suneel-ms

Copy link
Copy Markdown

Problem

#1826 deduplicates concurrent Syft runs per image in LinuxScanner. The first caller starts the shared run with its own CancellationToken, and later callers await that task. If the first caller is cancelled, the shared task faults and every caller waiting on it fails too.

LinuxContainerDetector and the experimental LinuxApplicationLayerDetector scan the same images at the same time. Experimental detectors are capped at 240 s (ExperimentalTimeoutSeconds, minus the 5 s buffer), whatever --Timeout is set to. When the experimental detector happens to start the shared run and Syft needs longer than ~235 s, its timeout cancels the scan for LinuxContainerDetector as well. That detector then reports no Linux components for the image, even though its own timeout hasn't expired.

Which detector starts the run is a race, so large images lose their Linux inventory intermittently:

  • Experiment starts the run: both Linux detectors log LinuxContainerDetector completed after at the same ~240–260 s, after Failed to run syft / OperationCanceledException, followed by Components skipped for Linux detector: <image>. The run still exits successfully.
  • Main detector starts the run: the experiment stops waiting at 235 s, and the main detector finishes later with a full inventory.

We hit this on large CUDA/ML images (roughly 4–11 GB compressed) whose Syft scans take 4–7 minutes. With --Timeout 2400, about half of those scans lost their Linux inventory, and the same build could keep one image's inventory and lose another's.

Fix

The shared run now has its own cancellation token, held by a small SharedSyftRun (caller count, result, token):

  • Every caller registers with the run and waits on its result with its own token. A caller that's cancelled only stops waiting.
  • When the last waiting caller gives up before the run finishes, the run is cancelled and its container stopped, so an abandoned scan doesn't keep running.
  • A caller arriving while a run is being torn down starts a fresh run instead of joining the cancelled one. Cache entries are removed by key and value, so a newer run is never evicted by an older run's cleanup.

Everything else is unchanged: one Syft container per image and parameters, errors from a scan still reach every waiting caller, completed entries are still removed, and a failed run can still be retried.

Testing

  • New LinuxScannerTests, both of which fail on main:
    • TestLinuxScanner_CancelledOwner_DoesNotCancelInFlightRunForOtherCallersAsync: the caller that starts the run is cancelled while another caller is waiting. The scan must keep running and the waiting caller must get the result, with only one container started.
    • TestLinuxScanner_AllCallersCancelled_CancelsInFlightSyftRunAsync: the container keeps running while any caller waits, is cancelled once all callers have given up, and a later caller starts a new run.
  • Microsoft.ComponentDetection.Detectors.Tests: all pass except TestVcpkgManifestFileAsync, which fails the same way on main on macOS (Windows path separators).
  • Microsoft.ComponentDetection.Orchestrator.Tests: 159/159 pass.
  • The LinuxScannerTests passed 25 consecutive runs.
  • Reproduced with real Docker and Syft on macOS arm64 against nvcr.io/nvidia/pytorch:26.01-py3, using --Timeout 2400. On main, one run ended with both Linux detectors cancelled at 236 s (Failed to run syft); in the other, the experiment stopped at 235 s and the main detector kept waiting. I couldn't verify the fix end to end on that image locally: Syft (~7.5 GB RSS) was OOM-killed in an 8 GB Docker VM, independently of this change.

Notes

  • Detector version: not bumped, following earlier Linux scanner reliability fixes such as Fix Docker scan hang: bound container removal, surface timeout telemetry #1778. Images affected by the race now report components they previously dropped, so I'm happy to bump it if you'd prefer.
  • Telemetry: if the caller that started a run gives up first, its LinuxScannerTelemetryRecord is disposed before the run finishes, so that scan's telemetry isn't recorded. Results are unaffected.
  • Narrower alternative: experimental detectors could be restricted to joining a shared run, never starting one. This change instead makes the shared run independent of any single caller, which also covers future callers with different timeouts. Happy to switch if you prefer the narrower fix.

LinuxScanner deduplicates concurrent Syft runs per image, but the first caller
ran the shared scan with its own cancellation token. When that caller was
cancelled, the scan failed for every caller waiting on it.

LinuxContainerDetector and the experimental LinuxApplicationLayer detector scan
the same images, and experimental detectors are capped at 240 seconds. When the
experiment started the shared run and Syft needed longer, its timeout cancelled
the scan for LinuxContainerDetector as well, which then reported no Linux
components for the image even though its own timeout had not expired. Which
detector started the run was a race, so large images lost their inventory
intermittently.

The shared run now has its own cancellation token. A cancelled caller only stops
waiting; the run is cancelled once no caller is left waiting for it, and a later
caller starts a new run instead of joining one that is being cancelled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 31, 2026 18:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

@suneel-ms

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="Microsoft"

@codecov

codecov Bot commented Oct 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.0%. Comparing base (d2b7149) to head (2a5394e).

Additional details and impacted files
@@     Coverage Diff      @@
##   main   #1906   +/-   ##
============================
============================

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants