Skip to content

fix(proxy): resolve upstream hostnames through trusted DNS - #18

Merged
cbhandsun merged 1 commit into
masterfrom
codex/upstream-dns-hardening
Sep 21, 2026
Merged

cbhandsun merged 1 commit into
masterfrom
codex/upstream-dns-hardening

Conversation

@cbhandsun

Copy link
Copy Markdown
Owner

Problem

The proxy intermittently returned 502 Bad Gateway to clients. The upstream
error was:

ERR_TLS_CERT_ALTNAME_INVALID: Host: chatgpt.com is not in the cert's altnames:
DNS:m.minghui.org, DNS:minghui.ca, ...

chatgpt.com was resolving to addresses belonging to an unrelated network, so
the TLS handshake legitimately failed certificate verification and the proxy
surfaced a gateway error.

Evidence

  • proxy-events.jsonl recorded 260 ERR_TLS_CERT_ALTNAME_INVALID events, in
    two clusters.
  • A request window of 106 completions contained 103 x 502 and 3 x 200, so the
    fault was intermittent rather than an outage.
  • A resolver test for chatgpt.com via the router (192.168.1.1) returned
    wrong addresses 40/40 times. 1.1.1.1, 8.8.8.8, 223.5.5.5 and
    192.168.99.1 all returned correct addresses.
  • The Windows DNS Client still intermittently selected the poisoned router
    answer, so the OS resolver chain is the failing layer.

Change

lib/proxy/upstream-dns.js adds a resolver that queries trusted servers
directly instead of the OS resolver chain, drops non-routable answers, and
keeps a bounded cache. Single-label and .local/.internal/.lan names still
use the system resolver, since public DNS cannot serve them.

lib/proxy/runtime/http-agents.js attaches that resolver to the outbound HTTPS
agent and exposes invalidateUpstreamAddress(hostname, errorCode), which drops
the cached answer only for failures that make the address itself suspect
(certificate mismatch, ECONNREFUSED, EHOSTUNREACH, ENETUNREACH). The
invalidator is attached to the agent so openai-proxy.js stays a composition
root within its line budget.

lib/proxy/upstream-retry.js classifies a failure as replayable when it
happened before any request body was transmitted. TLS buffers application data
until the handshake completes, so a reset during the handshake proves no HTTP
bytes reached the upstream. This is detected from connection state
(secureConnecting, reusedSocket) rather than the error code, so a reset on
an established or reused connection still requires the normal idempotency
contract and ECONNRESET is not broadly exempted.

appSettings.upstreamDnsServers selects the resolver list. It defaults on,
accepts an array or comma-separated string of up to 8 IPs, and an explicit
empty value disables the hardening. Invalid configuration now fails validation
instead of silently falling back to system resolution, which is the condition
that produced these 502s.

Verification

  • npm run verify passes in full, including lint, syntax, CI governance,
    release governance, coverage, frontend, integration, and Rust tests.
  • Added regression coverage: test/upstream-dns.test.js,
    test/upstream-retry-handshake.test.js (real sockets for handshake-phase
    resets, established-connection resets, and successful responses), plus
    extensions to test/upstream-retry.test.js,
    test/runtime-http-agents.test.js, and test/validator.test.js.
  • Live proof: with a simulated poisoned system resolver the resolver bypassed
    it entirely and completed a verified TLS handshake to chatgpt.com. After
    restarting the service, the post-restart window had 0
    ERR_TLS_CERT_ALTNAME_INVALID events and 0 handshake-phase ECONNRESET
    events, versus 260 and 93 respectively before.

Notes

The router at 192.168.1.1 still returns poisoned answers for chatgpt.com.
This change makes the proxy independent of it; other applications that use the
system resolver are unaffected by this fix.

Windows DNS Client intermittently answered chatgpt.com with addresses
belonging to an unrelated network, so the TLS handshake failed altname
verification and the proxy surfaced 502 to clients. Trusted resolvers
(1.1.1.1 / 8.8.8.8 / 223.5.5.5) returned correct addresses throughout,
which scopes the fault to the poisoned system resolver chain.

Query trusted servers directly instead of the OS resolver, reject
non-routable answers, and cache results briefly. Interrogate the
connection rather than the error code when deciding whether a failed
request may be replayed: TLS buffers application data until the
handshake completes, so a reset during the handshake proves no HTTP
bytes reached the upstream and replaying is side-effect free. Resets on
established or reused connections still require the normal idempotency
contract.

Certificate and unreachable-address failures additionally drop the
cached answer for that host so the retry re-resolves. The invalidator
lives on the agent to keep openai-proxy.js a composition root within its
line budget.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-21T15:49:03.995794Z e8f72a8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cbhandsun
cbhandsun merged commit 3f26851 into master Sep 21, 2026
16 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8f72a8f08

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/proxy/upstream-dns.js
Comment on lines +309 to +310
return withTimeout(
Promise.allSettled(queries).then((results) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve valid A answers when AAAA lookup times out

For the normal family: 0 lookup used by the HTTPS agent, this waits for both A and AAAA promises before inspecting either result. If a resolver returns a valid A record but its AAAA query is dropped or hangs, the outer timeout rejects the entire operation and discards the usable A answer; when this happens across the trusted resolvers, the code falls back to the potentially poisoned system resolver that this change is intended to bypass. Return the successful family once the other query times out rather than rejecting the aggregate result.

Useful? React with 👍 / 👎.

Comment thread lib/proxy/upstream-dns.js
Comment on lines +351 to +354
return Promise.all(attempts).then((results) => {
for (const item of results) {
if (item.result && item.result.addresses.length > 0) return item.result;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop waiting for irrelevant resolver attempts

When any configured resolver is unreachable, Promise.all waits for its full timeout even if an earlier-priority resolver has already returned a usable answer. Because the default list contains three public resolvers and cache entries expire after at most 30 seconds, environments that cannot reach just one of them incur roughly the 1.5-second query timeout on every cold lookup. Resolve as soon as the highest-priority successful result is known, or query the servers sequentially.

Useful? React with 👍 / 👎.

Comment on lines +71 to +75
if (env.PROXY_ALLOW_UNTRUSTED_UPSTREAM === '1') {
return {
rejectUnauthorized: false,
dnsConfig: upstreamDnsConfigFromConfig(lastKnownConfig),
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor DNS settings under the TLS environment override

When PROXY_ALLOW_UNTRUSTED_UPSTREAM=1, this branch never calls loadConfig, so lastKnownConfig remains null and the resolver always uses the built-in DNS servers. Consequently, an explicit upstreamDnsServers: [] cannot disable the feature and a custom resolver list is also ignored whenever the TLS environment override is enabled, which can break development or private-upstream setups that require system DNS. The environment override should affect certificate verification without bypassing the independent DNS configuration.

Useful? React with 👍 / 👎.

@cbhandsun
cbhandsun deleted the codex/upstream-dns-hardening branch September 21, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant