Repository navigation
fix(vtex): time out reviews extension calls so one stall can't hold the render - #1690
Draft
pedrofrxncx wants to merge 1 commit into
Draft
pedrofrxncx wants to merge 1 commit into
pedrofrxncx wants to merge 1 commit into
Conversation
…he render reviewsExt fires 2 reviews-and-ratings calls per product with no timeout and swallows their errors. A single stalled call on myvtex.com held whole page renders for ~55s before the swallowed failure let them return 200. Bound each call with AbortSignal.timeout(3s); on timeout the product just gets no rating, same as any other failure today.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Tagging OptionsShould a new tag be published when this PR is merged?
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
vtex/loaders/product/extend.ts→reviewsExtfires 2 calls per product to{account}.myvtex.com/reviews-and-ratings/...with no timeout, and swallows every error (.catch(() => ({}))). When one of those calls stalls, the page render waits for it, and then still returns200with no error logged. Since Oct 20 these stalls last ~55.5s and hit several VTEX storefronts.This PR bounds each reviews/rating call with
AbortSignal.timeout(3_000). On timeout the product gets no rating, which is exactly what happens today on any other failure, just without holding the render.Evidence
1. The stall is in this function
Uncached home render on
www.teciplast.com.br(apps 0.153.0), hitting the pod directly (bypassing Cloudflare and ingress) with deco debug timings (?__d=true):Product data is a cache hit (~80ms). The whole 55.5s is the
vtex/loaders/product/extensions/list.tsextension. One of the stalled shelves has onlyreviews: true(nobrands), so the call isreviewsExt.It also reproduces by invoking the loader in-process: 16 parallel
POST /live/invoke/vtex/loaders/product/extend.tswith{reviews: true}(17 products each, the same fan-out as one home render). Per round, 1–3 invocations took 55.8–56.2s while the rest finished in 1–2s.2. The hang is invisible
Because errors are swallowed, the stalled page returns
200after ~55s with nothing in the logs. Request latency distribution on teciplast (CloudflareOriginResponseDurationMs, 24h) has a sharp floor:3. It's upstream (VTEX reviews-and-ratings) and started ~Oct 20
Daily count of origin responses in the 55–56.5s band, per host:
The stalls are synchronized across these unrelated sites, some of which run on different clusters. They appear in the same hours: Oct 20 14–15h, Oct 21 15–16h, Oct 22 13–14h, then continuously from Oct 23 06h UTC. On most of them the stalls are on PDPs, where reviews are loaded. Teciplast is hit hardest because its home has 16 shelves with
reviews: true(~500 reviews calls per uncached render).What I ruled out
DNS (p99 246ms over 300 fresh connects from the pod), TCP (no stuck sockets / retransmits during a stall), the ingress and Cloudflare (reproduced against the pod directly), and VTEX search (
products.datawas a cache hit).What I could not pin down
None of that changes the fix: whatever the cause, an unbounded call to a decorative endpoint shouldn't be able to hold a page render.
Why 3s
Measured from the pod, healthy reviews-and-ratings calls are p95 ≈ 1.2s at 700 concurrent and < 0.7s at normal concurrency. 3s leaves headroom for slow-but-healthy responses while cutting the worst case from ~55s to ~3s. It's a constant, not a prop; it can become configurable if a store needs it.
Test
vtex/loaders/product/extend_test.tsstubs both endpoints with promises that only settle when theirsignalaborts:extend.ts): the test never settles (Promise resolution is still pending but the event loop has already resolved).deno fmt --checkanddeno lintpass on both files.Validation status
apps. Teciplast (apps 0.153.0) is being mitigated separately by removing the unused reviews extension: deco-sites/teciplast#244inventoryExthas the same no-timeout + swallow pattern (onvcs). Left out to keep this PR to what the evidence covers.Summary by cubic
Bounds the VTEX reviews extension calls with a 3s timeout so a single stalled reviews-and-ratings call can't hold a page render for ~55s.
Bug Fixes
AbortSignal.timeout(3_000); on timeout the product gets no rating, the same degradation as any other failure today.Written for commit f1b8caa. Summary will update on new commits.