[vpetersson-bot]

Issues Fixed

Sentry ANTHIAS-62 (OSError: [Errno 24] Too many open files) and ANTHIAS-63 (OperationalError: unable to open database file).

Description

Nothing bounds how many connections a client may open against an Anthias box, and the stock container descriptor limit is 1024. A burst walks the table to that ceiling, and the failure is not graceful: SQLite can no longer open the database, every request 500s, and the 500 handler cannot open 500.html either.

The device in Sentry was hit by a Qualys scan. The ordering there settles the cause: the Errno 24 events land at 13:39:21, the 371 database is locked events only from 13:39:22. Descriptor exhaustion came first — it is not downstream of the lock failures fixed by #3383, which closed a leak that made this worse but did not bound the burst.

Two guards, because neither is sufficient alone:

• Raise the soft descriptor limit to 8192. A process may raise its own soft limit up to the hard limit (the containers ship 1024/524288) without privilege, so this needs no ulimits: key — and therefore works identically under balena, whose supervisor renders its own compose file. Failure to raise is non-fatal; a headless device that will not boot is worse than one without the headroom.
• Bound concurrency. The descriptor raise alone only moves the ceiling, because every accepted connection runs the full Django path including a database connect. --limit-concurrency has uvicorn answer 503 as soon as the in-flight count is reached, costing no database handle and no template render. Default 256 — about six times the ~39 sockets a device holds while idle with the viewer polling — overridable with ANTHIAS_LIMIT_CONCURRENCY.

Applied to the development invocation as well: a dev stack on a routable address is as reachable as a fleet device.

Testing

Measured on the testbed with a 1500-connection burst, against a Pi 5 and a Pi 2 (slowest board, worst case for recovery):

	Pi 5	Pi 2
Unguarded	peaked ~930 fds, served unable to open database file out of get_new_connection	refused every connection past 1005, container died and was restarted by restart: always
Guarded	784 fds peak, cheap 503s, 0 descriptor/database errors	489 fds peak, process alive, 0 descriptor/database errors

Both boards stayed healthy afterwards. A 5000-connection burst against the guarded Pi 5 plateaued at 776 descriptors and still produced no errors.

Unit tests cover both guards, including that the development branch keeps its limit and that a refused descriptor raise cannot abort startup. Verified by mutation — removing either guard fails the suite.

Checklist

• I have performed a self-review of my own code.
• New and existing unit tests pass locally and on CI with my changes.
• I have done an end-to-end test for Raspberry Pi devices.
• I have tested my changes for x86 devices.
• I added a documentation for the changes I have made (when necessary).

Note: one pre-existing failure, tests/test_viewer.py::TestWaitForWaylandSocket::test_probe_reports_not_ready_for_a_non_socket_path, fails under pytest -n auto on master as well as here, and passes serially. Unrelated to this change and left alone.

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud