feat: add max-connection-age setting for HTTP/2 server connections - #1316
Conversation
Motivation: Long-lived HTTP/2 connections (as used by gRPC) lead to an uneven load distribution across server instances: clients stay connected to the instances they found at connect time, and instances added later (after a scale-out or a rolling deploy) receive no share of the existing traffic. The server is the side that can retire a connection gracefully, via GOAWAY. grpc-java offers this as maxConnectionAge; pekko-http has no equivalent (akka/akka-grpc#967 is the corresponding request on the Akka side). Modification: Add a `pekko.http.server.http2.max-connection-age` setting, default `infinite` (disabled). When a server connection reaches the configured age, the existing graceful termination path is triggered: GOAWAY(NO_ERROR) is sent, streams that are in flight complete normally, streams opened after the GOAWAY are refused with RST_STREAM(REFUSED_STREAM), and the connection is closed once no streams remain. The age is jittered per connection by a configurable fraction, `max-connection-age-jitter` (default 0.1 = +/- 10%, the value grpc-java applies; 0 disables jitter), so that connections that were opened together are not all closed at the same time. `triggerTermination` now accepts an infinite deadline, in which case no forced-close timer is scheduled. Also corrects the termination debug log, which printed the timer key instead of the deadline. Result: Operators can cap the lifetime of server-side HTTP/2 connections to rebalance long-lived connections across server instances. Behavior is unchanged by default. Tests: - sbt "http2-tests/test": 376 tests pass, including 2 new directional tests for max-connection-age in Http2ServerSpec - sbt validatePullRequest: passes - sbt "+http-core/mimaReportBinaryIssues": clean, with 5 new ReversedMissingMethodProblem filters for the added methods - sbt checkCodeStyle: clean; sbt headerCreateAll: no changes - sbt docs/paradox: builds, remaining warnings pre-existing - sbt sortImports: environment failure unrelated to this change (http/scalafixAll fails with NoSuchMethodError in scala.meta on a clean checkout of main too); the files changed here are sort-clean - manual end-to-end check against grpc-java 1.75.0: with max-connection-age = 5s, a client calling every 200 ms for 16 s saw 0 failures across 3 connection retirements, and a unary call that was in flight when the GOAWAY was sent completed normally References: None - no pekko-http issue tracks this; akka/akka-grpc#967 is the equivalent request against akka-http/akka-grpc.
|
1.4.1 is in RC - so use 1.4.2 here, replace the 2.0.0 values. Once this is merged, we can backport the change to the 1.4.x branch. |
| @@ -0,0 +1,23 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
| # or more contributor license agreements. See the NOTICE file | |||
There was a problem hiding this comment.
move to 1.4.x.backwards.excludes
There was a problem hiding this comment.
actually hold off on the version changes because I'm wondering if these chnages are too much for 1.x
There should be a 2.0.0 milestone release in the next few weeks.
There was a problem hiding this comment.
I just change the @SInCE annotations, but they can easily be changed back. Just let me know what you decide.
There was a problem hiding this comment.
@Kreinoee apologies but I had changed my mind - can we concentrate on this as a 2.0.0 change for now
1247 is a security hardening fix but this one seems more like a new feature
I would hope that we can get a new 2.0.0 milestone out in the next few weeks.
We can consider this for a future 1.5.0 release but I'd like to concentrate on getting it in 2.0.0 first.
There was a problem hiding this comment.
No worries, I will just change them back. But can I ask when 2.0.0 is estimated to get released. We are running 1.x in our production stack and have a need for this, so I would just like to know if I should look into an work arround on our own side until it is released.
And thanks for the super fast review, really appriciated. I am unfortunately out of time for today, and I have an all day meeting tommorrow, but I should have some time in the evening tomorrow, where I will have a look at you comments.
|
Thanks for this, the feature is useful and reusing the existing graceful-termination path is the right approach. The jitter handling and the corrected termination debug log look good. Scope: we'd like to treat this as a 2.0.0-only change for now. It is a new feature (new public settings and methods on Issues to address1. The drain after max-connection-age is unbounded ( Before this PR, every call to Please add a 2. A later
When already terminating, please (re)schedule 3. The Java API cannot express or round-trip "infinite" (
Minor (docs / comments)
|
|
I can't guarantee any release time lines. ASF requires us to have at least 3 PMC members review releases. We are struggling to get that this month. |
|
No pressure intended — I just wanted to know whether it's worth setting up a workaround on our side. A service mesh or L7 load balancer would be more than we need: our gRPC traffic is entirely in-cluster, and client-side load balancing paired with a max connection age is simple and works well for that case. The likely workaround is cherry-picking this onto my own fork of the 1.4.x branch and releasing it internally until an official release includes it. And yes, I'll keep helping to get this merged. |
This reverts commit 9ce450d.
…tionAge withMaxConnectionAge(getMaxConnectionAge) threw ArithmeticException on the default settings: toMillis overflows on ChronoUnit.FOREVER.getDuration, the value getMaxConnectionAge returns for an infinite age. Adds the inverse of JavaDurationConverter.toJava and a round-trip test.
The drain started by the MaxConnectionAge timer had no deadline, so a stream that never completes kept the connection open indefinitely. The new setting bounds it, with a default of 30s. The value infinite keeps the previous behaviour of waiting for all requests in flight to complete.
triggerTermination ignored every call after the first, so a server binding termination could not enforce its deadline on a connection that was already draining after max-connection-age. Now an earlier deadline reschedules the forced close, a later one is ignored. The max-connection-age timer is cancelled once any termination starts, so the age and its grace period never shorten a termination in progress.
…rmination starts Same behaviour, but the rule is visible where it applies: the timer handler checks whether a termination is already in progress and logs that there is nothing to do, rather than relying on the cancelled timer's message being dropped by the stage.
…ng, next to the age setting
…jitter test comment
…letion-timeout setting The message is shared by both sides. On the client the deadline is the completion-timeout setting, on the server it is the deadline passed to terminate or the max-connection-age-grace setting.
|
I looked into your comments now. Some I implemented as you suggested, and on some I pushed back a little: Scope: understood, 2.0.0 it is. The 1. Unbounded drain. Added One deliberate deviation from your suggestion: 2.
I figured that this behaviour is easy to document, and therefore easy for users to reason about: the documentation of 3. Java API round trip. Added Minor: both done. The HTTP/2-only note is in reference.conf and the docs, and the test comment now says what the assertion checks rather than tightening the margin. One extra: while running the grace period against a real grpc-java client I noticed that the server-side forced-close log message pointed at Verification after the changes: http2-tests 380/380 (4 new tests), |
|
Thanks @Kreinoee for the thorough follow-up. All the points from the earlier review are addressed, and the explanations were really helpful. Keeping A few small points, none of them blocking:
|
|
@Kreinoee I'm hoping to produce an RC for 2.0.0-M2 soon. The nits above, would you time to look at them. If not, I might just merge this and add a follow up PR later. |
pjfanning
left a comment
There was a problem hiding this comment.
lgtm - I will follow up with a new PR to handle some nits
…n setters Motivation: Getters such as ConnectionPoolSettings.getKeepAliveTimeout return ChronoUnit.FOREVER.getDuration for an infinite value, but the matching Java setters converted with toScala, which throws IllegalArgumentException for that value. Passing a getter's result back to its setter failed on the default settings of keep-alive-timeout, max-connection-lifetime and periodic-keep-alive-max-idle. Modification: Use JavaDurationConverter.toScala, the inverse of the toJava used by the getters, in the Java setters of ClientConnectionSettings.idleTimeout, ConnectionPoolSettings idleTimeout, keepAliveTimeout, maxConnectionLifetime and responseEntitySubscriptionTimeout, and WebSocketSettings.periodicKeepAliveMaxIdle. Result: Infinite durations round-trip through the Java API. Tests: - sbt "http-core/testOnly org.apache.pekko.http.scaladsl.settings.*": 27 pass; the 3 new round-trip tests fail without the fix - sbt "http-core/mimaReportBinaryIssues": clean References: Refs apache#1316
Motivation: Twenty commits landed on main after the model was pinned to `40b07a2`. Two touch a claim it makes. apache#1264 closes the one P1 gap the model recorded as open: the HTTP/2 frame parser now rejects a frame over `max-frame-size` (512kB) on its frame header, before buffering the payload. apache#1296 validates the two application-supplied parts of the request line at construction, which changes what the §11 `Raw-Request-URI` misuse can reach. The rest do not move a claim: apache#1316 adds `max-connection-age`, default `infinite`; apache#1301 encodes HPACK literals as ISO-8859-1, which still substitutes '?' above 0xFF, and the P2 guard checks the rendered bytes; apache#1305, apache#1298, apache#1318 and the dependency updates are not security-relevant. Modification: Re-pin to `478c58b`. Add `max-frame-size` to §5a, §6 and the §15 back-map, add apache#1264 to the P1 verification paragraph, and drop the "one P1 gap is open" paragraph. Restate the §11 `Raw-Request-URI` misuse: injection is now rejected at construction, what remains is client bytes choosing an unnormalized target, and a value that passes construction yet corrupts the request line is `VALID` under §5b.4. Add the matching §15 row. Result: The model is verified against `478c58b` and records no open P1 gap. Tests: Not run - docs only References: Refs apache#1264, apache#1296
|
Super, thanks. And sorry I did not follow up on you nits, I just had a busy week. I hope I can maybe do it in a follow up next week, where I can also make one for the other fix than I promised. |
#1323 is open for the nits but if you see anything else that is needed, feel free to make suggestions |
…ER round-trip in Java setters (#1323) * Follow-up to max-connection-age review: document and test infinite grace with a later terminate Motivation: Review of #1316 noted that reference.conf only documents one direction of the interaction between max-connection-age and a server binding termination, that the combination of max-connection-age-grace = infinite with a later terminate was not tested, and that some new tests had tight timing margins. Modification: - reference.conf: state that a later terminate with an earlier deadline shortens a drain started by the age, also with an infinite grace period, mirroring http2.md. - Http2ServerSpec: add a test for max-connection-age-grace = infinite followed by terminate(10.millis). - Http2ServerSpec: widen timing margins (age 1s vs expectNoBytes(700ms), terminate(2s) vs expectNoBytes(700ms)). Result: The documented and tested behaviour covers both directions, and the timing sensitive tests have more headroom on loaded CI machines. Tests: - sbt "http2-tests/testOnly ...Http2ServerSpec -- -z max-connection-age": 7 tests pass - scalafmt on the changed spec References: Refs #1316 * Map ChronoUnit.FOREVER back to Duration.Inf in the other Java duration setters Motivation: Getters such as ConnectionPoolSettings.getKeepAliveTimeout return ChronoUnit.FOREVER.getDuration for an infinite value, but the matching Java setters converted with toScala, which throws IllegalArgumentException for that value. Passing a getter's result back to its setter failed on the default settings of keep-alive-timeout, max-connection-lifetime and periodic-keep-alive-max-idle. Modification: Use JavaDurationConverter.toScala, the inverse of the toJava used by the getters, in the Java setters of ClientConnectionSettings.idleTimeout, ConnectionPoolSettings idleTimeout, keepAliveTimeout, maxConnectionLifetime and responseEntitySubscriptionTimeout, and WebSocketSettings.periodicKeepAliveMaxIdle. Result: Infinite durations round-trip through the Java API. Tests: - sbt "http-core/testOnly org.apache.pekko.http.scaladsl.settings.*": 27 pass; the 3 new round-trip tests fail without the fix - sbt "http-core/mimaReportBinaryIssues": clean References: Refs #1316
…che#1316 Motivation: apache#1316 merged with `infinite` as the disabled value for `max-connection-age`, a `Duration` setting type and `JavaDurationConverter` for the Java API. The client setting used `0s` and `FiniteDuration`. Modification: - `persistent-connection-max-age` defaults to `infinite`, is a `Duration`, and must be > 0 or `infinite`, as on the server - Java accessors use `JavaDurationConverter`, so `ChronoUnit.FOREVER.getDuration` round-trips to `Duration.Inf` - reference.conf, scaladoc and docs follow the server wording - the jitter scheduling matches `Http2Demux` - settings tests move to a new `Http2ClientSettingsSpec`, mirroring `Http2ServerSettingsSpec` - MiMa excludes are reduced to the abstract members that need them - pekko-style imports in the changed files Result: The client and server max connection age settings share naming conventions, defaults, validation and Java conversion behaviour. Tests: - sbt "http-core/testOnly ...Http2ClientSettingsSpec ...Http2CommonSettingsSpec ...Http2ServerSettingsSpec": 16 passed - sbt "http2-tests/testOnly ...Http2PersistentClient*": 26 passed - sbt "http-core/mimaReportBinaryIssues": clean - sbt scalafmtAll and headerCreateAll run on changed modules References: Refs apache#1319, apache#1316
* http2: add persistent client connection max age #1319 Motivation: Long-lived managed HTTP/2 clients can stay pinned to existing server instances. Modification: Add a configurable maximum age that retires managed persistent HTTP/2 connections after in-flight requests drain. Result: Requests arriving after retirement establish a fresh connection while in-flight requests complete normally. Tests: - sbt validatePullRequest - sbt http2-tests/test - sbt +http-core/mimaReportBinaryIssues - sbt scalafmtCheckAll scalafmtSbtCheck - sbt +headerCheckAll - sbt docs/paradox - sbt checkCodeStyle - git diff --check References: Refs #1319 * http2: address max-age review feedback Expose client max-age and jitter as public settings, add per-connection jitter, and preserve a buffered request when the request source completes during retirement. Document break-before-make behavior and extend retirement/reconnect coverage. * http2: preserve Java max-age precision * Align client max connection age with the server-side setting from #1316 Motivation: #1316 merged with `infinite` as the disabled value for `max-connection-age`, a `Duration` setting type and `JavaDurationConverter` for the Java API. The client setting used `0s` and `FiniteDuration`. Modification: - `persistent-connection-max-age` defaults to `infinite`, is a `Duration`, and must be > 0 or `infinite`, as on the server - Java accessors use `JavaDurationConverter`, so `ChronoUnit.FOREVER.getDuration` round-trips to `Duration.Inf` - reference.conf, scaladoc and docs follow the server wording - the jitter scheduling matches `Http2Demux` - settings tests move to a new `Http2ClientSettingsSpec`, mirroring `Http2ServerSettingsSpec` - MiMa excludes are reduced to the abstract members that need them - pekko-style imports in the changed files Result: The client and server max connection age settings share naming conventions, defaults, validation and Java conversion behaviour. Tests: - sbt "http-core/testOnly ...Http2ClientSettingsSpec ...Http2CommonSettingsSpec ...Http2ServerSettingsSpec": 16 passed - sbt "http2-tests/testOnly ...Http2PersistentClient*": 26 passed - sbt "http-core/mimaReportBinaryIssues": clean - sbt scalafmtAll and headerCreateAll run on changed modules References: Refs #1319, #1316 --------- Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com>
|
Cool, that looks good to me, so nothing left for me to do. Thanks for the review and help. |
Motivation
Long-lived HTTP/2 connections (as used by gRPC) lead to an uneven load distribution across server
instances: clients stay connected to the instances they found at connect time, and instances added
later (after a scale-out or a rolling deploy) receive no share of the existing traffic. The server
is the side that can retire a connection gracefully, via GOAWAY. grpc-java offers this as
maxConnectionAge; pekko-http has no equivalent (akka/akka-grpc#967 is the corresponding requeston the Akka side).
Modification
Add a
pekko.http.server.http2.max-connection-agesetting, defaultinfinite(disabled). When aserver connection reaches the configured age, the existing graceful termination path is triggered:
GOAWAY(NO_ERROR) is sent, streams that are in flight complete normally, streams opened after the
GOAWAY are refused with RST_STREAM(REFUSED_STREAM), and the connection is closed once no streams
remain.
The age is jittered per connection by a configurable fraction,
max-connection-age-jitter(default0.1 = +/- 10%, the value grpc-java applies; 0 disables jitter), so that connections that were opened
together, for example after a deploy, are not all closed at the same time.
triggerTerminationnow accepts an infinite deadline, in which case no forced-close timer isscheduled. Also corrects the termination debug log, which printed the timer key instead of the
deadline.
Result
Operators can cap the lifetime of server-side HTTP/2 connections to rebalance long-lived
connections across server instances. Behavior is unchanged by default.
Verified against a real grpc-java (1.75.0) client: with
max-connection-age = 5son a pekko-httpserver, a client issuing a unary call every 200 ms for 16 s observed 0 failures across 3 connection
retirements (the server saw 4 distinct client connections), and a unary call that was in flight when
the GOAWAY was sent completed normally on the connection being drained.
Tests
sbt "http2-tests/test": 376 tests pass, including 2 new directional tests formax-connection-age in
Http2ServerSpec(GOAWAY(NO_ERROR) + connection close on an idleconnection; in-flight stream completes and late stream is refused)
sbt "+http-core/mimaReportBinaryIssues": clean, with 5 newReversedMissingMethodProblemfilters for the added methods on
Http2ServerSettings(@ApiMayChange/@DoNotInherit) and theinternal
Http2Demuxsbt scalafmtCheckAll scalafmtSbtCheck: cleansbt headerCreateAll: no changessbt docs/paradox: builds (remaining warnings are pre-existing and unrelated)sbt validatePullRequest: passessbt sortImports: environment failure unrelated to this change (http/scalafixAllfails withNoSuchMethodErrorin scala.meta on a clean checkout ofmaintoo); the files changed here aresort-clean
References
None - no pekko-http issue tracks this; akka/akka-grpc#967 is the equivalent request against
akka-http/akka-grpc.