Skip to content

fix(vtex): time out reviews extension calls so one stall can't hold the render - #1690

Draft
pedrofrxncx wants to merge 1 commit into
mainfrom
fix/vtex-reviews-ext-timeout
Draft

pedrofrxncx wants to merge 1 commit into
mainfrom
fix/vtex-reviews-ext-timeout

Conversation

@pedrofrxncx

@pedrofrxncx pedrofrxncx commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

vtex/loaders/product/extend.ts → reviewsExt fires 2 calls per product to {account}.myvtex.com/reviews-and-ratings/... with no timeout, and swallows every error (.catch(() => ({}))). When one of those calls stalls, the page render waits for it, and then still returns 200 with no error logged. Since Oct 20 these stalls last ~55.5s and hit several VTEX storefronts.

This PR bounds each reviews/rating call with AbortSignal.timeout(3_000). On timeout the product gets no rating, which is exactly what happens today on any other failure, just without holding the render.

Evidence

1. The stall is in this function

Uncached home render on www.teciplast.com.br (apps 0.153.0), hitting the pod directly (bypassing Cloudflare and ingress) with deco debug timings (?__d=true):

load-data;dur=55522
pages-home…@sections.variants.1.value.6.products.data;desc=hit;dur=85
pages-home…@sections.variants.1.value.6.products.extensions.0;desc=bypass;dur=560      ← normal
pages-home…@sections.variants.1.value.16.products.data;desc=hit;dur=83
pages-home…@sections.variants.1.value.16.products.extensions.0;desc=bypass;dur=55518  ← stalled

Product data is a cache hit (~80ms). The whole 55.5s is the vtex/loaders/product/extensions/list.ts extension. One of the stalled shelves has only reviews: true (no brands), so the call is reviewsExt.

It also reproduces by invoking the loader in-process: 16 parallel POST /live/invoke/vtex/loaders/product/extend.ts with {reviews: true} (17 products each, the same fan-out as one home render). Per round, 1–3 invocations took 55.8–56.2s while the rest finished in 1–2s.

2. The hang is invisible

Because errors are swallowed, the stalled page returns 200 after ~55s with nothing in the logs. Request latency distribution on teciplast (Cloudflare OriginResponseDurationMs, 24h) has a sharp floor:

bucket (ms) requests
54000–55250 3
55250–55500 76
55500–56000 2135
56000–56500 128

3. It's upstream (VTEX reviews-and-ratings) and started ~Oct 20

Daily count of origin responses in the 55–56.5s band, per host:

host Oct 14–18 Oct 19–23 Oct 23 only
www.teciplast.com.br 201 3357 2361
www.todolivro.com.br 28 512 394
www.alphabeto.com 32 411 300
www.taiff.com.br 10 185 141
www.happybooks.com.br 3 163 132
www.lojainterdental.com.br 30 145 114
www.palacio.com.br 2 121 101

The stalls are synchronized across these unrelated sites, some of which run on different clusters. They appear in the same hours: Oct 20 14–15h, Oct 21 15–16h, Oct 22 13–14h, then continuously from Oct 23 06h UTC. On most of them the stalls are on PDPs, where reviews are loaded. Teciplast is hit hardest because its home has 16 shelves with reviews: true (~500 reviews calls per uncached render).

What I ruled out

DNS (p99 246ms over 300 fresh connects from the pod), TCP (no stuck sockets / retransmits during a stall), the ingress and Cloudflare (reproduced against the pod directly), and VTEX search (products.data was a cache hit).

What I could not pin down

  • I couldn't capture what VTEX actually returns on the stalled call (status/body), because the error is swallowed.
  • A standalone Deno process in the same pod, firing up to 700 concurrent reviews/rating calls (including after idle gaps up to 90s), never stalled. Inside the running app it does.
  • So there may be an additional connection-reuse factor in the long-lived process.

None of that changes the fix: whatever the cause, an unbounded call to a decorative endpoint shouldn't be able to hold a page render.

Why 3s

Measured from the pod, healthy reviews-and-ratings calls are p95 ≈ 1.2s at 700 concurrent and < 0.7s at normal concurrency. 3s leaves headroom for slow-but-healthy responses while cutting the worst case from ~55s to ~3s. It's a constant, not a prop; it can become configurable if a store needs it.

Test

vtex/loaders/product/extend_test.ts stubs both endpoints with promises that only settle when their signal aborts:

  • With this change: passes in ~3s, products come back with an empty rating.
  • Without it (stashed extend.ts): the test never settles (Promise resolution is still pending but the event loop has already resolved).

deno fmt --check and deno lint pass on both files.

Validation status

  • Unit test (above), and confirmed it fails on current code
  • Real-environment validation: sites pick this up only after bumping apps. Teciplast (apps 0.153.0) is being mitigated separately by removing the unused reviews extension: deco-sites/teciplast#244
  • Not changed: inventoryExt has the same no-timeout + swallow pattern (on vcs). Left out to keep this PR to what the evidence covers.

Summary by cubic

Bounds the VTEX reviews extension calls with a 3s timeout so a single stalled reviews-and-ratings call can't hold a page render for ~55s.

Bug Fixes

  • Old code fired both reviews-and-ratings calls per product with no timeout and swallowed their errors, so one stalled call held the render ~55s and still returned 200.
  • Each call now passes AbortSignal.timeout(3_000); on timeout the product gets no rating, the same degradation as any other failure today.
  • Adds a unit test that stubs both endpoints to hang until aborted, verifying products resolve with an empty rating within the timeout.

Written for commit f1b8caa. Summary will update on new commits.

View guided diff Turn on auto-fix

…he render

reviewsExt fires 2 reviews-and-ratings calls per product with no timeout
and swallows their errors. A single stalled call on myvtex.com held whole
page renders for ~55s before the swallowed failure let them return 200.
Bound each call with AbortSignal.timeout(3s); on timeout the product just
gets no rating, same as any other failure today.
@coderabbitai

coderabbitai Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Tagging Options

Should a new tag be published when this PR is merged?

  • 👍 for Patch 0.165.5 update
  • 🎉 for Minor 0.166.0 update
  • 🚀 for Major 1.0.0 update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant