Repository navigation
Don't let one caller's cancellation abort a shared Syft run - #1906
Suneel Freimuth (suneel-ms) wants to merge 1 commit into
Conversation
LinuxScanner deduplicates concurrent Syft runs per image, but the first caller ran the shared scan with its own cancellation token. When that caller was cancelled, the scan failed for every caller waiting on it. LinuxContainerDetector and the experimental LinuxApplicationLayer detector scan the same images, and experimental detectors are capped at 240 seconds. When the experiment started the shared run and Syft needed longer, its timeout cancelled the scan for LinuxContainerDetector as well, which then reported no Linux components for the image even though its own timeout had not expired. Which detector started the run was a race, so large images lost their inventory intermittently. The shared run now has its own cancellation token. A cancelled caller only stops waiting; the run is cancelled once no caller is left waiting for it, and a later caller starts a new run instead of joining one that is being cancelled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
|
@microsoft-github-policy-service agree company="Microsoft" |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1906 +/- ##
============================
============================
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Problem
#1826 deduplicates concurrent Syft runs per image in
LinuxScanner. The first caller starts the shared run with its ownCancellationToken, and later callers await that task. If the first caller is cancelled, the shared task faults and every caller waiting on it fails too.LinuxContainerDetectorand the experimentalLinuxApplicationLayerDetectorscan the same images at the same time. Experimental detectors are capped at 240 s (ExperimentalTimeoutSeconds, minus the 5 s buffer), whatever--Timeoutis set to. When the experimental detector happens to start the shared run and Syft needs longer than ~235 s, its timeout cancels the scan forLinuxContainerDetectoras well. That detector then reports no Linux components for the image, even though its own timeout hasn't expired.Which detector starts the run is a race, so large images lose their Linux inventory intermittently:
LinuxContainerDetector completed afterat the same ~240–260 s, afterFailed to run syft/OperationCanceledException, followed byComponents skipped for Linux detector: <image>. The run still exits successfully.We hit this on large CUDA/ML images (roughly 4–11 GB compressed) whose Syft scans take 4–7 minutes. With
--Timeout 2400, about half of those scans lost their Linux inventory, and the same build could keep one image's inventory and lose another's.Fix
The shared run now has its own cancellation token, held by a small
SharedSyftRun(caller count, result, token):Everything else is unchanged: one Syft container per image and parameters, errors from a scan still reach every waiting caller, completed entries are still removed, and a failed run can still be retried.
Testing
LinuxScannerTests, both of which fail onmain:TestLinuxScanner_CancelledOwner_DoesNotCancelInFlightRunForOtherCallersAsync: the caller that starts the run is cancelled while another caller is waiting. The scan must keep running and the waiting caller must get the result, with only one container started.TestLinuxScanner_AllCallersCancelled_CancelsInFlightSyftRunAsync: the container keeps running while any caller waits, is cancelled once all callers have given up, and a later caller starts a new run.Microsoft.ComponentDetection.Detectors.Tests: all pass exceptTestVcpkgManifestFileAsync, which fails the same way onmainon macOS (Windows path separators).Microsoft.ComponentDetection.Orchestrator.Tests: 159/159 pass.LinuxScannerTestspassed 25 consecutive runs.nvcr.io/nvidia/pytorch:26.01-py3, using--Timeout 2400. Onmain, one run ended with both Linux detectors cancelled at 236 s (Failed to run syft); in the other, the experiment stopped at 235 s and the main detector kept waiting. I couldn't verify the fix end to end on that image locally: Syft (~7.5 GB RSS) was OOM-killed in an 8 GB Docker VM, independently of this change.Notes
LinuxScannerTelemetryRecordis disposed before the run finishes, so that scan's telemetry isn't recorded. Results are unaffected.