Skip to content

Fix flake from interrupted remote channel shutdown - #9

Open
4genealogic-filmset wants to merge 1 commit into
9.2.0-with-backported-fixesfrom
ssmith/fix-channel-shutdown-interruption
Open

4genealogic-filmset wants to merge 1 commit into
9.2.0-with-backported-fixesfrom
ssmith/fix-channel-shutdown-interruption

Conversation

@4genealogic-filmset

Copy link
Copy Markdown

An interrupt received while Bazel waits for a remote gRPC channel to terminate is wrapped in IOException, then promoted to AssertionError by ReferenceCountedChannel. That kills the Bazel server and flakes the build with Server terminated abruptly.

Finish channel shutdown through interruptions, preserve the existing total timeout, and restore the thread's interrupt status afterward. The existing protection only handles an interrupt already present when shutdown begins.

Four of six matching infrastructure failures in the 72-hour audit show this cause chain. The other two socket closures have no confirmed cause and are outside this fix's claimed coverage.

Based on deployed Bazel 9.2.9993 (449456745b83d5ec3e270af623e0b9cc189d5a3e), targeting 9.2.0-with-backported-fixes. Rollout requires publishing and adopting the next fork release.

Validation:

  • Before the fix, both single and repeated mid-wait interrupt regression tests fail with the same exception seen in CI.
  • After the fix, all four channel shutdown tests pass.
  • The full remote gRPC suite passes: 90 tests, zero failures or errors.
  • Formatted both Java files with google-java-format; git diff --check passes.

Test command: bazel test //src/test/java/com/google/devtools/build/lib/remote/grpc:grpc --noshow_progress --test_output=errors.

@blainefreestone blainefreestone left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine Bazel maintainers could be more opinionated about this but for our purposes this is definitely good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants