RHCLOUD-49985 - Revert hummingbird migration to fix latency issues - #240
Merged
EvanCasey13 merged 1 commit intoAug 5, 2026
Conversation
This reverts the hummingbird migration (commit bd94932 and subsequent fixes) to restore the stable OpenResty 1.21.4 configuration that was in place before July 10, 2026. Root Cause Analysis: - On July 10, hummingbird migration upgraded OpenResty from 1.21.4 (nginx 1.21) to 1.27.1 (nginx 1.27) - a 6 major version jump - On July 14 at 4:55 PM, SLO alerts fired: * Gateway: 89.88% of requests slower than 1000ms * Entitlements: 84.66% of requests slower than 1000ms * UHC Auth Proxy: Latency SLO breach - Emergency fixes attempted July 15 (OpenResty 1.29 bump, library fixes, buffering changes) all failed to resolve the issue The OpenResty 1.27.1 upgrade introduced breaking nginx behavior changes: - Different proxy buffering defaults - Stricter timeout enforcement (lua tcp socket read timeouts) - Changed upstream keepalive/connection pool behavior - HTTP/2 directive deprecation Changes in this revert: - Dockerfile: Restore pre-hummingbird configuration * Base image: hi/core-runtime:2.43 → ubi9/ubi-minimal:9.8-1782797275 * OpenResty: 1.27.1-1.el9 → 1.21.4-1.el8 * Remove multi-stage build complexity (library clobber issue) * Restore jaegertracing, openresty-resty, openresty-opentracing packages - gateway/http.d/core.conf: Remove client_body_buffer_size symptom fix Hummingbird migration deferred until OpenResty compatibility is resolved. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
EvanCasey13
changed the title
|
Hi there, APIcast maintainer here, sorry for comment on merged PR but I don't see any button to open new issue. I see you are still using 1.21.4-1. I recommend upgrading to version 1.21.4-4, which includes several CVE patches. Ping me if you need more details Regarding your upgrade to 1.27.1
By setting
Do you have details on this, if you open an issue upstream I will see if I can fix it.
You would need to cherry-pick this PR 3scale#1577 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://redhat.atlassian.net/browse/RHCLOUD-49985
This reverts the hummingbird migration (commit bd94932 and subsequent fixes) to restore the stable OpenResty 1.21.4 configuration that was in place before July 10, 2026.
Root Cause Analysis:
The OpenResty 1.27.1 upgrade introduced breaking nginx behavior changes:
Changes in this revert:
Hummingbird migration deferred until OpenResty compatibility is resolved.