Silent crash, when DNS on bootstrap fails ? #205

Activity

1. aarond10 commented on Jul 26, 2026

   

   aarond10

   on Jul 26, 2026

   Owner

   More actions

   #207 is looking at what to do when the underlying network interface changes. If restarting the daemon in full isn't an option, we can explore SIGHUP1 or similar. Will close this as a dup.
2. aarond10

   closed this as a duplicateof Slow/hung DoH requests after a WAN flap until the process is restarted — proposing in-process recovery (netlink self-heal and/or SIGUSR1) #207 on Jul 26, 2026
3. alen127 commented on Aug 13, 2026

   

   alen127

   on Aug 13, 2026

   Last edited by alen127

   More actions

   I think this issue should be reopened. I got the same fatal dns_poller.c:39 error shown in the attached https-dns-proxy.txt, but it happened during normal operation and not at startup.

   I had gpt 5.6 sol investigate, pasting it here in case it is of some help:

   Investigation

   Environment:

      - OpenWrt 24.10.2
      - https-dns-proxy 2026.05.06-r1
      - c-ares 1.34.6
      - libcurl 8.19.0, mbedTLS 3.6.6, nghttp2 1.63.0
      - Two proxy instances:
        - Quad9 with two IPv4 bootstrap servers
        - Cloudflare Security with two IPv4 bootstrap servers
      - Bootstrap polling interval was the default 120 seconds
   
   

   The proxies had been operating normally for approximately two days. Then both independently entered crash loops:

   09:39:12 [F] dns_poller.c:39 c-ares needed more IO event handler, than the number of provided nameservers: 2
   ...
   09:39:51 procd: Instance https-dns-proxy::instance2 is in a crash loop, 6 crashes
   
   09:45:37 [F] dns_poller.c:39 c-ares needed more IO event handler, than the number of provided nameservers: 2
   ...
   09:46:16 procd: Instance https-dns-proxy::instance1 is in a crash loop, 6 crashes
   

   procd eventually stopped respawning both instances, leaving dnsmasq with no working upstream servers. Manually restarting https-dns-proxy immediately restored DNS.

   There was a WAN DHCP lease renewal shortly beforehand, but:

   • the same WAN address was retained;
   • there was no carrier/link loss;
   • the router was not under memory or file-descriptor pressure;
   • no OOM or kernel errors occurred.

   Issue #207 appears related but describes stale libcurl/TCP connections after a WAN flap. This failure is specifically a fatal exit in dns_poller.c when c-ares requests more simultaneous socket watchers than io_events_count.

   The current code allocates:

     d->io_events = calloc(nameservers, sizeof(ev_io));
     d->io_events_count = nameservers;

   and exits when c-ares requests another socket:

     if (!io_event_ptr) {
         FLOG("c-ares needed more IO event handler...");
     }

   The number of simultaneously required c-ares sockets does not appear guaranteed to be less than or equal to the configured nameserver count, particularly during retries, concurrent A/AAAA resolution, or transport fallback.
4. kubrickfr commented on Sep 22, 2026

   

   kubrickfr

   on Sep 22, 2026

   More actions

   Seconding @alen127's request to reopen, and flagging that #213 already fixes this — it just needs a review pass to land.

   I hit this today and it reproduces deterministically on demand, so it isn't an occasional mystery. Posting the reproduction as independent confirmation, plus the impact argument, since this is currently closed as a dup of #207 and I don't think it is one.

   Not a dup of #207

   #207 is about libcurl's connection pool holding stale TLS/TCP connections after a WAN flap, causing slow queries. It says so explicitly:

   The c‑ares bootstrap resolver uses fixed IPs, so it mostly keeps working — the hang is essentially all in the curl connection pool.

   This bug is in the other half of the daemon: a fatal exit(1) from the c-ares bootstrap poller in dns_poller.c. Different file, different library, different symptom. Fixing #207 will not fix this, and a WAN flap is only one of the ways to trigger it — any bootstrap DNS timeout will do.

   Root cause

   dns_poller_init() sizes the watcher array by counting commas in -b (dns_poller.c#L251-L265):

   unsigned nameservers = 1;
   for (unsigned i = 0; bootstrap_dns[i]; i++)
       if (bootstrap_dns[i] == ',') nameservers++;
   d->io_events = (ev_io *)calloc(nameservers, sizeof(ev_io));
   d->io_events_count = nameservers;

   and FLOGs (i.e. exit(1)) the moment c-ares asks for one socket more than that.

   The implicit invariant is "c-ares never has more open sockets than configured servers". That is false for c-ares >= 1.32, which changed to a connection-based model. On a query timeout, c-ares does not close the connection — it retires it and opens a fresh one for the retry:

   • process_timeouts() sets ARES_CONN_FLAG_NONEW on the connection that timed out (c-ares 1.34.8 ares_process.c:800). The socket stays open.
   • ares_fetch_connection() then refuses to reuse a NONEW connection (ares_process.c:1315), so the retry opens a new socket.
   • The retired socket is only reclaimed later, once idle, by ares_check_cleanup_conns() (ares_close_sockets.c:119).

   So open sockets transiently exceed the server count by design. With POLLER_QUERY_TRIES 4 and a parallel A + AAAA lookup, the peak is driven by retry rounds, not by server count. Anyone using two bootstrap servers has exactly two slots and loses.

   Reproduction

   Build as packaged, point -b at silently-dropped addresses (TEST-NET), and run:

   https_dns_proxy -r https://dns.nextdns.io/<id> -p 5999 -a 127.0.0.1 \
       -b 198.51.100.1,198.51.100.2 -vvvv
   

   Annotated debug log — dead in 1.5 seconds:

   [D] 1790076447.415672 dns_poller.c:41 Reserved new io event: 0x7fb1998dc0   <- conn1 -> server1
   [D] 1790076448.115730 dns_poller.c:175 Processing DNS query timeouts
   [D] 1790076448.115862 dns_poller.c:41 Reserved new io event: 0x7fb1998df0   <- conn1 retired, conn2 -> server2
   [D] 1790076448.216372 dns_poller.c:31 Released used io event: 0x7fb1998dc0  <- conn1 reclaimed 100ms LATER
   [D] 1790076448.817085 dns_poller.c:41 Reserved new io event: 0x7fb1998dc0   <- conn2 retired, conn3
   [D] 1790076448.917559 dns_poller.c:175 Processing DNS query timeouts
   [F] 1790076448.917683 dns_poller.c:39 c-ares needed more IO event handler, than the number of provided nameservers: 2
   

   Note the third line: the new socket is registered 100 ms before the retired one is released, which is precisely the overlap the array has no room for.

   With a single -b entry it dies on the first timeout. Padding the same two distinct servers to six -b entries (repetition is enough — the count is computed from the raw string) survived 60 s of total bootstrap blackhole with 27 socket churns. That is a usable stopgap for anyone stuck on a released build, but it only widens the margin; the bound is retry rounds, not entries.

   Impact

   This is worse than a plain crash, because of how it interacts with procd. The OpenWrt init script uses a bare procd_set_param respawn, i.e. threshold 3600 / timeout 5 / retry 5. Each crash happens in ~1.5 s, so every respawn counts as "too fast" and after five attempts procd stops respawning permanently. dnsmasq is left with no working upstream and DNS stays down until someone intervenes; manual restarts also die instantly while the bootstrap servers are still unreachable. In my case it took a reboot. @alen127 saw procd give up on both instances the same way, and the trigger there was an ordinary WAN DHCP lease renewal that kept the same address.

   So a routine PPPoE re-dial or lease renewal can take DNS down indefinitely, with nothing in the logs after the reboot wipes them.

   Environment

   • https-dns-proxy 2026.05.06-r1 (0ba0525), OpenWrt, aarch64
   • c-ares 1.34.8, libcurl 8.21.0 / mbedTLS 3.6.6 / nghttp2 1.70.0, libev 4.33
   • Reproduced with 2 bootstrap servers, default 120 s polling interval

   @alen127 hit it on c-ares 1.34.6, so this spans c-ares releases.

   On #213

   For anyone landing here first: #213 is the fix, and having read it, it takes the right approach. Allocating individually owned watcher nodes on demand is necessary rather than stylistic — ev_io_start() retains the watcher pointer, so growing a contiguous array with realloc() would leave libev holding freed addresses. It also replaces the fd = 0 free marker with ARES_SOCKET_BAD, which closes a second latent problem in the same function: with 0 as the sentinel, get_io_event(d, 0) cannot distinguish "free slot" from "slot legitimately holding fd 0", so a live watcher could be aliased and re-ev_io_init()'d while active. That is unreachable under procd (fd 0 is /dev/null, I checked) and under daemon(0,0), but it is the kind of thing that only stays unreachable by luck.

   Given that #213 exists, is reviewed, and carries a regression test that fails on unpatched upstream, the main thing missing here is reopening this issue so the two are linked and the severity is on record.