Repository navigation
Conversation
…cation, setup retries DeepSeek answers several endpoints with HTTP 200 carrying a business-level error (code=40003 INVALID_TOKEN, "messages too frequent", killed web session) instead of a 4xx. The old code indexed straight into data.biz_data, so those responses died with `TypeError: 'NoneType' object is not subscriptable` inside a route, and because no status was attached the token-rotation path never fired: a dead token stayed pinned and every later turn failed the same way in ~330 ms. functions.py - Add UpstreamBizError(UpstreamError) and _biz_data(); auth-shaped business messages map to 401 (quarantine/rotate), everything else to 429 (cool down and rotate). Replace all raw biz_data indexing in create_challange_pow, create_new_chat, upload_file and get_file_content. Existing rotation in handle_chat picks it up with no changes because the status rides on UpstreamError. app.py - _setup_failure_outcome / _setup_retry_kwargs: create_new_chat and create_challange_pow run before the try block that owns rotation, so an auth failure there escaped handle_chat as a bare 500 and the dead token was never marked. Classify those, mark the token, and re-enter handle_chat with the token excluded — OUTSIDE the per-signature asyncio.Lock, which is not reentrant and previously hung the request. - POST /v1/messages/count_tokens (plus /messages/count_tokens): the route did not exist, so every Claude Code context probe returned 404. - _with_deadline: a hard ceiling on upstream work. ClientTimeout has total=None and sock_read resets on keepalive bytes, so a slowly-dripping upstream never tripped anything (the access log has 19.7-minute /v1/messages entries). - _is_truncated_stream_error: upstream hang-ups mid-stream logged a full traceback and left the client on a dead socket. Downgrade to a warning and keep the terminal error frame. tests - test_upstream_hardening.py: 15 cases for count_tokens, _biz_data mapping, the deadline wrapper and truncation detection. - test_setup_retry.py: 2 cases for the pre-turn auth failure path, including that the retry does not deadlock on the signature lock. 274 tests pass.
CONTRIBUTING requires a CHANGELOG bullet and README update in the same PR as new user-visible behavior. count_tokens is now served, and upstream business errors surface as 401/429 instead of a 500 traceback.
|
Hi @Rrainich — thanks for this PR, the biz-error hardening is much needed. We just debugged a live incident tonight (2026-10-31 ~23:24 CST) that fits exactly the failure shape you describe (HTTP 200, ~330 ms instant failure), and we found the root cause. Sharing here since it interacts with this PR: Upstream protocol change:
Fix verified live in our fork 1515asd/rh-deeperseeker (commit Your |
Upstream began rejecting parent_message_id=0 with biz_code 26 "invalid message id" on 2026-10-31. The rejection arrives as an application/json envelope on the SSE endpoint, so the parser saw zero `data:` lines and raised the generic "empty stream" 502 — no cause, no traceback. - Omit parent_message_id entirely on the first turn instead of sending 0 or null. Edges disagree about null; omission is the only shape that works everywhere. Follow-up turns still chain the stored parent id. - Keep the non-SSE body tail and the response content-type, and log both when a stream ends with no output. A JSON error envelope in that tail is promoted into the raised UpstreamError message.
da775b1 to
75fcfdb
Compare
What this fixes
DeepSeek often answers HTTP 200 with a business-level error (
code=40003 INVALID_TOKEN, "messages too frequent", a killed web session) rather than a 4xx. The bridge indexed straight intodata.biz_dataon those responses, which raisedTypeError: 'NoneType' object is not subscriptableinside the route. No status was attached, so the token-rotation path never fired — the dead token stayed pinned in the pool and every later turn failed the same way in ~330 ms until it was removed by hand.Changes
functions.pyUpstreamBizError(UpstreamError)+_biz_data()— one helper fordata.biz_data. Auth-shaped business messages (login|auth|token|session|expire) map to 401 so the token is quarantined/rotated; everything else maps to 429 so it cools down and rotates. Because the status rides onUpstreamError,_upstream_http_code()and the existing 401/403/429 path inhandle_chatpick it up unchanged.["biz_data"]index:create_challange_pow,create_new_chat,upload_file(×2),get_file_content(×2).app.py_setup_failure_outcome/_setup_retry_kwargs—create_new_chat/create_challange_powrun before the try block that owns rotation, so an auth failure there escapedhandle_chatas a bare 500 and the dead token was never marked. These now classify the failure, mark the token, and re-enterhandle_chatwith the failed token excluded — outside the per-signatureasyncio.Lock, which is not reentrant (recursing inside it hung the request).POST /v1/messages/count_tokens(plus the/messages/count_tokensalias) — the route did not exist, so every Claude Code context probe was a 404._with_deadline— the sharedClientSessionusesClientTimeout(total=None, connect=15, sock_read=600), andsock_readresets on every keepalive byte, so a slowly-dripping upstream never tripped anything (the access log has 19.7-minute/v1/messagesentries). This puts a configurable ceiling (DEEPSEEKER_REQUEST_DEADLINE, default 600 s) on upstream work and raisesUpstreamError(504). Scope note: it is applied inside the SSE generators because they outlivehandle_chat's return. It covers "upstream goes silent" and cancels the upstream generator; a slow client still surfaces asCancelledErrorrather than a 504, so it is not an end-to-end hard ceiling._is_truncated_stream_error— upstream hang-ups mid-stream (aiohttp.ClientPayloadError/TransferEncodingError) logged a full traceback and left the client on a dead socket. Downgraded to a warning; the terminal error frame still goes out.Tests
tests/test_upstream_hardening.py— 15 cases:count_tokensauth + flattening,_biz_datastatus mapping, deadline pass-through / 504 on hang / no deadlock on early abort, truncation detection.tests/test_setup_retry.py— 2 cases: a 401 oncreate_new_chatrotates and succeeds instead of 500-ing, and the retry does not deadlock on the signature lock.Behaviour change to be aware of
A flagged/invalid token now produces an honest 401/429 and rotation instead of a wall of 500s. That is recovery, not repair: if every token belongs to the same flagged account, rotation has nowhere to go and the client sees a 429. That is the correct signal — the previous 500 loop was hiding it.
Not covered
hardening-backup.patchin my working tree is a local rollback artifact and is not part of this PR.