Repository navigation
fix: backport controller, snapshot, router and read recovery fixes to 1.8 - #3742
Merged
Merged
Conversation
Parquet embeds relocated Jackson classes in `parquet-jackson`, so upgrading the regular Jackson modules does not update that copy. This matters for [CVE-2026-110879](https://www.cve.org/CVERecord?id=CVE-2026-110879) / [GHSA-m8xf-2cr7-77ph](https://github.com/advisories/GHSA-m8xf-2cr7-77ph), whose remediation requires Jackson databind 2.18.12 or later. `main` already selects `parquet-jackson:1.18.0`, containing Jackson core/databind 2.22.1 and annotations 2.22, independently of Parquet 1.15.2 and Iceberg 1.6.1. Add compatibility coverage for that combination and document why the separate pin is needed. Add an S3Unit test that writes and reads Parquet records through Iceberg, including escaped strings and a nullable field. Serialize the resulting footer through Parquet's compact and pretty JSON paths, then parse both with shaded Jackson and verify the metadata and row count. Expose the existing runtime Parquet APIs to test compilation without adding runtime dependencies. Explain the existing shaded-Jackson constraint and add its embedded Jackson versions to the binary license inventory. (cherry picked from commit 13c80c8)
…ery (#3723) When every unfenced broker session expires, controller overload protection cancels the deferred stale-broker fencing check. Timed-out heartbeats can refresh a broker session and complete overload recovery, but the cancelled check remained absent. Other expired brokers could stay unfenced until a normal heartbeat or registration arrived. Resume stale-broker fencing when the overload circuit breaker moves from `HALF_OPEN` to `OPEN`. This makes recovery-driven scheduling work for both normal and timed-out heartbeats. The breaker notifies its active controller at the recovery transition. The controller registers this callback when it creates an active heartbeat manager, then uses its existing fencing scheduler to restore the earliest broker-session check. The callback runs after the heartbeat manager updates the recovered broker's session, so the scheduler sees the current broker state. --------- Co-authored-by: codex <noreply@openai.com> (cherry picked from commit c61ff32)
Snapshot replicas are updated outside the ordinary partition-stop path. Removing or replacing one can leave an existing delayed Fetch waiting until its long-poll timeout, although the old replica can no longer serve it. Recheck delayed Fetch and Produce requests when a snapshot replica changes identity. An unchanged replica does not trigger completion. Publish the new snapshot-replica mapping before rechecking delayed requests. Run the recheck outside the map's compute callback to avoid introducing a map/purgatory lock-order inversion. Existing request eligibility and error handling decide whether the request completes. Regression coverage exercises removal, replacement with a fenced replica, publication before completion, and an unchanged mapping that must leave the Fetch waiting. Local verification passed: both snapshot-transition regression tests, Checkstyle and Java formatting. The repository's `rat` task was invoked and skipped by upstream configuration. ```bash ./gradlew --build-cache core:S3UnitTest --tests 'unit.kafka.server.streamaspect.ElasticReplicaManagerTest.testSnapshot*' ./gradlew --build-cache rat checkstyleMain checkstyleTest spotlessJavaCheck ``` Start with `ElasticReplicaManager.computeSnapshotReadPartition`, then the snapshot-transition tests in `ElasticReplicaManagerTest`. This change has no prerequisite in this series. Related to #3669. AI assistance: OpenAI Codex (GPT-6) was used to prepare the contribution split, adapt the changes to main, and update the tests and descriptions. (cherry picked from commit c1a37a1)
## Why MultiBucketsRouterChannel started a full batch interval whenever it opened a new LingerBatch. After a size-triggered flush, this shifted the next timed flush later than the cadence used by the object WAL writer. RemoteProxy also scheduled a callback that read its mutable current batch rather than the batch that owned the timer. Its size limit counted epoch keys, so many requests for one epoch could stay in an oversized batch. Together these behaviors made router batching less predictable and produced more proxy RPCs than necessary. ## What Align MultiBuckets timed flushes with the object WAL writer's force-flush schedule. Each new batch derives its delay from the planned flush time, which survives a size-triggered flush. Bind each RemoteProxy timer to its own batch. The callback ignores obsolete batches and seals the active batch after 100 us of idle time or 1 ms from its creation, whichever comes first. A send seals a batch only when it reaches 8,192 actual requests, regardless of epoch grouping. Sealed batches remain queued until an in-flight permit is available. ## Reviewer notes Review MultiBucketsRouterChannel's force-flush schedule first, then RouterOutV2.RemoteProxy's batch ownership and timer path, followed by the RequestBatch deadline and size tests. (cherry picked from commit fcde982)
## Why `AbstractObjectStorage.rangeRead` wraps its future with a two-minute `timeoutWithNewReturn`. If a retryable object storage failure keeps the underlying read in its retry loop beyond that deadline, the wrapper fails the caller's future even though retries are still ongoing. For WAL reservation reads, `ObjectReservationService.verify()` maps that timeout to `false`, and `DefaultWriter` treats it as another node fencing the WAL. This can turn a transient storage outage into a broker shutdown. ## What Remove the two-minute timeout wrapper from `rangeRead` so retryable reads can continue through the existing retry policy until they succeed or fail with a non-retryable error. Non-retryable errors continue to propagate through the existing failure handling. ## How Return the original range-read future. Remove the timeout callback that released a ByteBuf if the underlying read completed after its caller-facing timeout; with the wrapper gone, the original future owns completion and buffer delivery directly. Closes #3724 ## Reviewer notes Review `AbstractObjectStorage.rangeRead` together with `mergedRangeRead0`: the range-read future remains governed by the existing retry classification and scheduling. (cherry picked from commit f942648)
## Why A block-cache read failure can abort coordinator offset loading even when a subsequent read could succeed. Retrying inside the same StreamReader is insufficient when the reader's own state prevents recovery. ## What Retry any failed StreamReaders read once after a ten-second asynchronous delay, using a fresh reader and the original stream, offset range, and byte limit. If the second attempt fails, propagate its exception to the caller. ## How The failed attempt closes its reader before scheduling the retry. The retry discards any cached reader at the requested offset and creates a new one; successful attempts retain the existing reader-cache behavior. The retry decision emits an ERROR with its cause and delay; existing cache failure logging is preserved. Scheduler rejection completes the caller's future exceptionally. Regression coverage exercises recovery from an arbitrary failure, the one-retry limit, scheduler rejection, and another read completing on the same cache event loop during the retry delay. (cherry picked from commit e508d57)
Collaborator
Author
|
Validation:
|
|
|
Collaborator
Author
|
CI completed successfully: run 41959700119.
The separate |
Gezi-lzq
approved these changes
Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bring selected compatibility and reliability fixes to the 1.8 release line. Controller overload recovery can leave stale brokers unfenced, snapshot replica transitions can strand delayed fetches, router batching can exceed its intended bounds, and a failed block-cache read can require a fresh reader to recover.
What
How
Preserve the source fixes and their regression coverage. Adapt dependency declarations, snapshot test imports to the existing 1.8 APIs without introducing unrelated main-branch functionality. Each backport commit records its source SHA.