fix(http): honor content-type charset before encoding detection (graphon #93) - #41252
Open
wylswz wants to merge 1 commit into
Open
fix(http): honor content-type charset before encoding detection (graphon #93)#41252wylswz wants to merge 1 commit into
wylswz wants to merge 1 commit into
Conversation
The HTTP request node's Response.text guessed the encoding with charset_normalizer first and ignored the charset declared in the Content-Type response header. Auto-detection can misjudge UTF-8 as another encoding, producing garbled Chinese text. Honor the declared charset first, then fall back to charset_normalizer and httpx. Manual backport of graphon PR #93 (the graph engine is vendored in api/dify_graph on lts/1.13.x, not the external graphon package). Public record: dify #31695. (manually backported from graphon PR langgenius/graphon#93) Resolves ENG-767
dosubot
Bot
added
the
size:S
Contributor
Pyrefly Type Coverage
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport of graphon PR #93 ("honor content-type charset before encoding detection") to
lts/1.13.x.On
lts/1.13.xthe graph engine is still vendored in-repo (api/dify_graph/), so this is applied as a manual port rather than a bumpedgraphondependency (which is howmaincarries the fix).Root cause: the HTTP request node's
Response.textrancharset_normalizerdetection first and ignored thecharsetdeclared in the responseContent-Typeheader. Auto-detection can misjudge UTF-8 as another encoding, producing garbled Chinese output — even though the server declared the correct charset.Fix: honor the declared
Content-Typecharset first (decode witherrors="replace"), then fall back tocharset_normalizerand finally httpx's built-in detection when no charset is declared. Added_extract_charset_from_content_type()usingemail.message.Message.get_content_charset()so quoted parameters are handled correctly.Public record: dify #31695.
Test plan
test_entities.py: declared charset takes precedence over detection, GBK real-world case,errors="replace"on invalid bytes, and charset nested in quoted params is ignored.Response.texttests still pass (40 passed).ruff check/ruff format/basedpyrightclean on changed files.Resolves ENG-767