[client] Back off writes rejected by disk protection

[fhan688]

Purpose

Linked issue: #4278

When disk protection rejects writes with DISK_WRITE_LOCKED, the Java client can immediately retry and repeatedly resend the full payload. During sustained disk pressure, this generates excessive network traffic and consumes client CPU without making progress.

This PR introduces bounded exponential backoff for Java Log and KV writers. Queued writes targeting a rejected bucket wait before being sent again, while other eligible buckets continue to make progress. Writes retry automatically when the backoff expires, subject to the existing retry budget.

This addresses the Java-client portion of the issue. Rust-client support is planned in a separate PR.

Brief change log

Add two client configuration options:

• client.writer.disk-write-locked.backoff: initial backoff for writes rejected by disk protection; defaults to 1s.
• client.writer.disk-write-locked.backoff-max: maximum backoff, including jitter; defaults to 10s.

The delay doubles with the existing batch retry count and uses ±20% jitter. Configuration must satisfy 1ms <= initial <= maximum <= 2147483647ms. Setting both options to the same value selects a fixed delay without jitter.

Implement the following scheduling and retry behavior:

• Handle disk rejection in the shared Sender retry path, covering both RPC-level failures and per-bucket errors for ordinary and historical Log/KV writes. Install backoff only when the batch remains eligible for retry.
• Track monotonic deadlines by the actual destination TableBucket. Historical writes sharing a physical destination share its backoff. Concurrent updates retain the longer remaining deadline.
• Enforce backoff in both readiness checks and batch draining. Keep disk backoff independent of KV backpressure and wait for the longer remaining duration. Late successful responses and KV pressure updates do not clear an active disk deadline.
• Preserve existing retry limits, writer IDs, batch sequences, payload ownership, and historical re-enqueue behavior. Flush and graceful close respect backoff and the existing close timeout.
• Let Sender wait until the earliest eligible retry instead of spinning, including when client.writer.batch-timeout is zero.
• Wake Sender after re-enqueueing, periodically remove expired deadlines, and clear state during final resource destruction.

Tests

Validation used JDK 11.0.27 and Maven 3.8.6.

Passed checks

• All 342 client unit tests, including RecordAccumulatorTest (25 tests) and SenderTest (51 tests).
• All 3 tests in the new DiskWriteBackoffITCase, covering real-cluster Log/KV rejection and recovery, successful reads after recovery, and the close timeout.
• Spotless, Apache RAT, configuration documentation generation, and git diff --check.
• Checkstyle 9.3 CLI checks using the repository's Checkstyle and suppression configurations.

Coverage includes exponential growth and capping, jitter, deadline boundaries, concurrent updates and cleanup, bucket isolation, KV backpressure interaction, both error delivery paths, historical destinations, idempotence, retry exhaustion, and Sender waiting and wakeup.

Local traffic comparison

The comparison used one writer, one bucket, one TabletServer, replication factor one, a 4096-byte string per single-row batch, zero batch timeout, and approximately five seconds of simulated disk protection. The patched client used the default backoff settings.

• Log writes: write RPCs decreased from 73,794 to 3, and payload bytes decreased from 306,835,452 to 12,474. The patched client completed the pending write approximately 1.61 seconds after protection was lifted.
• KV writes: write RPCs decreased from 75,404 to 3, and payload bytes decreased from 311,870,944 to 12,408. The patched client completed the pending write approximately 1.50 seconds after protection was lifted.

Recovery time measures the interval from lifting protection to the successful write callback. Payload counters exclude transport overhead. These are single local samples demonstrating reduced repeated sends; they do not establish production capacity or the absence of a normal-throughput regression.

Full-suite verification remains outstanding

• Client verify encountered a rolling-upgrade health assertion failure and metadata-update errors in FlussLakeTableITCase and PartitionedTableITCase. It subsequently stalled during FlussTableITCase.testAppendAndPoll and was stopped after collecting a thread dump.
• Isolated reruns of the three failing methods passed on both the baseline and patched implementations. The append-and-poll test also passed separately for all three record formats. These results do not resolve the full-suite failures.
• Five ReplicaFetcherThreadTest cases were rejected by disk protection because actual test-host disk usage was approximately 97.3%, above the 85% threshold.
• The Maven Checkstyle plugin failed to load commons-collections.FastHashMap. Maven runs therefore used -Dcheckstyle.skip=true, with source validation performed separately through the Checkstyle CLI as described above.

API and Format

Adds two Java-client configuration options. Public method signatures, RPC messages, error codes, and storage formats are unchanged.

The implementation uses the existing DISK_WRITE_LOCKED error code and requires upgrading the Java client or the connector that bundles it; no server protocol upgrade is required.

Backoff applies per physical table bucket. Requests already in flight may still complete, and this mechanism does not impose a cluster-wide bandwidth limit. After protection is lifted, writes may wait for the remaining backoff window before retrying.

Documentation

Update website/docs/maintenance/configuration.md with the two configuration options and their defaults, validation constraints, backoff behavior, interaction with KV backpressure, flush and close semantics, recovery delay, and client upgrade requirements.

Generate the option descriptions through the existing configuration documentation generator and synchronize them into the maintenance configuration page.

[swuferhong]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

[fhan688]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

I'd like to confirm that extracting an internal WriteThrottleController in this PR or as a follow-up PR?

[swuferhong]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

I'd like to confirm that extracting an internal WriteThrottleController in this PR or as a follow-up PR?

I'd prefer to do it in this PR: set up the architecture properly upfront and reduce the amount of refactoring needed later. WDYT @platinumhamburg

[platinumhamburg]

Hi, @fhan688 I suggest extracting an internal WriteThrottleController to unify the existing disk-write backoff and KV backpressure handling:

• Policies translate errors or pressure signals into a throttling reason, scope, and deadline.
• The controller maintains state separately for each reason, computes the effective deadline, handles cleanup, and wakes the Sender when the next eligible send time moves earlier.
• ready() queries the remaining delay, and drain() checks eligibility again before draining batches. Adding a new policy should not require changes to either path.
• If we later need QPS or bandwidth limits, we can introduce non-blocking permit acquisition before sending requests. If the goal is simply to suppress aggressive retries during write rejection, we could allow only a bounded number of probe requests after the backoff expires.

This would make disk-write backoff one policy within a reusable write-throttling mechanism and provide a natural extension point for future throttling scenarios.

I'd like to confirm that extracting an internal WriteThrottleController in this PR or as a follow-up PR?

I'd prefer to do it in this PR: set up the architecture properly upfront and reduce the amount of refactoring needed later. WDYT @platinumhamburg

Agree +1.