feat: add client-side max connection age for HTTP/2

[Rayan-and-beyond]

Motivation

Long-lived managed HTTP/2 clients can remain pinned to existing server instances, leaving newly added instances underused after a scale-out or rolling deployment.

Modification

Add two HTTP/2 client settings:

• pekko.http.client.http2.persistent-connection-max-age (default infinite, disabled)
• pekko.http.client.http2.persistent-connection-max-age-jitter (default 0.1, i.e. +/-10%)

Both are exposed through the Scala and Java Http2ClientSettings APIs. Jitter semantics and validation match the server-side setting in #1316: >= 0, < 1, and 0 disables jitter.

When a managed persistent connection reaches its jittered age, it stops accepting new requests and drains in-flight requests before disconnecting. Retirement is currently break-before-make: a buffered/new request is held until the old connection disconnects, then a replacement connection is established. If completion-timeout expires, remaining in-flight requests on the retiring connection are terminated.

The reconnect path preserves a buffered request even when the request source completes during retirement. The timer now tracks the current Connected state directly rather than using a callback stored at stage level.

Result

Operators can periodically refresh long-lived managed HTTP/2 client connections without synchronized reconnect spikes, while the current break-before-make latency/timeout trade-off is documented explicitly.

Tests

Fresh validation after review changes:

• sbt "http-core/testOnly org.apache.pekko.http.scaladsl.settings.Http2ClientSettingsSpec" ^T 9 passed
• full Http2PersistentClientPlaintextSpec ^T 13 passed, including the normal failure/reconnect paths
• TLS retirement tests ^T 2 passed
• sbt "+http-core/mimaReportBinaryIssues" ^T pass on Scala 2.13.18 and 3.3.8
• sbt scalafmtCheckAll scalafmtSbtCheck ^T pass
• sbt docs/paradox ^T pass (only pre-existing duplicate-anchor warnings)
• git diff --check ^T pass

References

Fixes #1319

[pjfanning]

The server-side equivalent (#1316) applies a jitter to the max connection age (pekko.http.server.http2.max-connection-age-jitter, default 0.1, i.e. each connection is closed after 90%–110% of the configured age, same as grpc-java). Could this PR do the same on the client side, e.g. pekko.http.client.http2.persistent-connection-max-age-jitter?

Without jitter, a fleet of clients that connected at roughly the same time (which is exactly the situation after a scale-out or rolling deployment) will all retire and reconnect at the same moment, causing a synchronized reconnect spike on the servers. With the retirement being break-before-make, it would also cause a correlated latency spike across the clients.

It would be good to keep the client and server settings consistent in naming, semantics and validation (jitter >= 0 and < 1, 0 disables it) with #1316.

[pjfanning]

Thanks @Rayan-and-beyond for picking this up — the change is nicely contained, and it's great to see docs and tests included. A few further concerns beyond the jitter one above:

1. The setting shouldn't go through internalSettings

persistent-connection-max-age is a public config key, but it's carried as Some(Http2PersistentConnectionSettings(...)) in Http2ClientSettings.internalSettings, which is intended as a placeholder for internal things like custom strategies. Consequences:

• Anyone calling withInternalSettings(...) with some other internal setting silently loses the max age (the collect in PersistentConnection.apply then falls back to Duration.Zero).
• There's no programmatic API (persistentConnectionMaxAge / withPersistentConnectionMaxAge) and no javadsl counterpart, unlike every other key in that config block.
• internalSettings changes from None to always Some(...) by default.

Could this be a proper field on Http2ClientSettings (scaladsl + javadsl), with a MiMa excludes file? maxHeaderListSize in 1.4.1 (http-core/src/main/mima-filters/1.4.x.backwards.excludes/http2-max-header-list-size.excludes) and the server-side #1316 are good precedents.

2. A buffered request can be dropped if upstream completes while retiring

During retirement the replacement InHandler keeps one pushed request in the requestIn slot and ignores onUpstreamFinish. If the user pushes one more request and then completes the request source, once the old connection drains onDisconnected sees isClosed(requestIn) and calls completeStage(). The inlet reports closed while the un-grabbed element is still in the slot, so that request never gets dispatched or answered. I think onDisconnected needs to check isAvailable(requestIn) before completing, and reconnect to send it. A test for "request pushed after the age timer fires, then upstream completes" would cover this.

3. Retirement is break-before-make, which should at least be documented

Once retirement starts, new requests are backpressured until all in-flight streams finish or completion-timeout (default 3s) forcibly closes the old connection, and then a new connection (TCP + TLS + H2 handshake) is established. So:

• every retirement adds the drain time plus the connection setup time to new requests' latency;
• long-lived streams (gRPC server-streaming, SSE, large downloads) are cut off after completion-timeout. The docs say completion-timeout "bounds how long an in-flight request may delay retirement", but don't make it clear that those requests are terminated.

Make-before-break (open the new connection, move dispatch to it, let the old one drain) would avoid both issues. If that's out of scope for this PR, please spell out the trade-off in reference.conf and http2.md.

4. Smaller points

• maxConnectionAgeCallback: Option[() => Unit] at stage level is only a way to reach Connected.retire() from onTimer. Tracking the current Connected state directly would be simpler.
• The changes to Unconnected.onPull and the onDisconnected reconnect branch (connect when requestIn already holds an element) also affect the normal failure/reconnect path, not just retirement. It would be worth mentioning that in the PR description, and possibly adding a targeted test.
• The new test relies on tight timings (300ms age, expectNoRequest(500.millis) / expectNoRequest(100.millis)), which may be flaky on slow CI. There's also no test for retiring an idle connection (no in-flight requests), then reconnecting lazily on the next request.

[Rayan-and-beyond]

Thanks, these were good catches. Addressed in 30e52c4:

• added persistent-connection-max-age-jitter with the same default/validation semantics as feat: add max-connection-age setting for HTTP/2 server connections #1316
• moved max age + jitter onto the public Scala/Java Http2ClientSettings API and added MiMa excludes; internalSettings is back to None by default
• fixed the closed+buffered inlet case so a request queued during retirement is still sent even if the request source completes
• documented the break-before-make behavior, added latency/long-lived-stream caveats, and clarified completion-timeout
• replaced the stage-level callback with direct tracking of the current Connected state
• widened the retirement timing margins and added idle-retirement/lazy-reconnect coverage

The buffered-request regression initially exposed one additional closed-inlet pull edge in the replacement connection; that is fixed too. Fresh validation is green: settings spec 3/3, full plaintext persistent-client spec 13/13 (including normal reconnect paths), TLS retirement 2/2, MiMa on Scala 2.13/3.3, scalafmt, docs, and git diff --check. CI is rerunning on the pushed commit.

[pjfanning]

Thanks for the quick update @Rayan-and-beyond, having a proper public setting with a Java API is much better.

One suggestion on the Java duration conversions. The new getter/setter round-trip through milliseconds (copying the existing getCompletionTimeout/withCompletionTimeout lines):

def getPersistentConnectionMaxAge: Duration = Duration.ofMillis(persistentConnectionMaxAge.toMillis)
def withPersistentConnectionMaxAge(maxAge: Duration): Http2ClientSettings =
  self.withPersistentConnectionMaxAge(maxAge.toMillis.millis)

This truncates sub-millisecond precision. Since 0 means "disabled" for this setting, a tiny positive age passed from Java would silently disable the feature. java.time.Duration.toMillis also throws an ArithmeticException for very large values.

scala.jdk.DurationConverters (standard library in Scala 2.13 and 3, already used in http-core, e.g. ConnectionPoolSettings, ClientConnectionSettings, ServerBinding) converts losslessly in both directions:

import scala.jdk.DurationConverters._

def getPersistentConnectionMaxAge: java.time.Duration = persistentConnectionMaxAge.toJava
def withPersistentConnectionMaxAge(maxAge: java.time.Duration): Http2ClientSettings =
  self.withPersistentConnectionMaxAge(maxAge.toScala)

If the setting ever needs to support infinite, pekko.http.impl.util.JavaDurationConverter should be used instead: its toJava maps Duration.Inf to ChronoUnit.FOREVER.getDuration, and #1316 adds the inverse toScala. On that note, #1316 uses infinite to disable the server-side max-connection-age, while this PR uses 0s. It may be worth aligning the two once #1316 is merged.

The existing completionTimeout conversions have the same issue but can be left as they are in this PR.

[Rayan-and-beyond]

Good point. I switched the new Java max-age accessor/mutator to scala.jdk.DurationConverters so the conversion is lossless and does not depend on toMillis. I also added a 1ns round-trip regression test.

Validation after the change:

• Http2CommonSettingsSpec: 3/3
• scalafmt checks: pass
• +http-core/compile: pass on Scala 2.13.18 and 3.3.8
• git diff --check: pass

I left the disabled value as 0s for now because #1316 is still open and #1319 explicitly calls out 0s = disabled (default). Happy to align it to infinite once the server-side shape lands, or sooner if maintainers prefer that in this PR.

[pjfanning]

@Rayan-and-beyond I merged #1316 - would you be able to rebase this and try to maintain consistency with it?

[pjfanning]

I added 3c81498 to bring this more in synch with #1316

[pjfanning]

lgtm

[pjfanning]

I merged this to get into 2.0.0-M2-RC1. If there are any issues with this PR, it can still be reworked in a new PR. Milestone releases don't mean that the APIs and behaviours need to be maintained for backward compatibility.