Skip to content

feat: block scanner junk at the nginx edge - #2884

Merged
mroderick merged 2 commits into
masterfrom
feature/scanner-junk-edge-block
Sep 17, 2026
Merged

mroderick merged 2 commits into
masterfrom
feature/scanner-junk-edge-block

Conversation

@mroderick

@mroderick mroderick commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

Block scanner junk at the nginx edge so it never reaches Rails.

Analysis of the canonical log archive (2026-09-16) showed 91% of ChapterController requests are 404s from automated scanning: sensitive-file probes (SSH keys, cloud configs, .env files), Vite RSC source-map probing, and path guesses landing on the /:id chapter catch-all. The app already answers every one with a 404; this change stops them at nginx instead, with client-visible behaviour unchanged.

Changes

Two regex locations in config/nginx.conf.erb, ahead of the location / proxy:

  • Sensitive or unknown file extensions (nested paths included), excluding the real asset dirs (/assets, /uploads, /static, /tom-select, /packs) and the advertised /sitemap.xml.gz
  • Extension-less probes seen in the log archive: id_rsa-style key names, wp-* paths, /__vite_rsc_findSourceMapURL, the z9x8c7v6b5-debug-trigger-codebar.io probe, Dockerfile, __debug__, userfiles, backup, private[-_]key

New CI workflow (.github/workflows/nginx.yml) renders the ERB, runs nginx -t, and asserts a curl matrix: scanner probes 404 at the edge, legitimate paths (including /sitemap.xml.gz, chapter slugs, and the exact-match Plausible proxy paths) pass through.

Verification

  • Rendered config passes nginx -t
  • Full probe/passthrough matrix verified against a live nginx:alpine container (60+ probe paths 404; 20+ legitimate paths unchanged)
  • Every blocked pattern returned 404 from the app before this change, so no client-visible behaviour change
Review notes
  • The asset-dir exclusion list is a hand-maintained allowlist. A future public/ asset directory outside the five excluded prefixes would silently 404 its files at the edge; the new CI smoke matrix is the guard, so add new public dirs to both the lookahead and the matrix.
  • The .gz exclusion is by exact filename (sitemap.xml.gz) rather than dropping gz from the extension list, so backup-archive probes (site.tar.gz, db.sql.gz) stay blocked.
  • Known status-code drift: format-suffixed resource routes (e.g. /sponsors.json) return an edge 404 where the app returned 406. Only scanner-style probes hit those; accepted as part of the behaviour-unchanged scope.
  • Chapter slugs colliding with blocklist keywords (e.g. a chapter named backup) would 404 at the edge. No current slug collides; the chapter-slug namespace is not checked automatically.
  • Out of scope here: the auth app gets the same treatment in feat: block scanner junk at the nginx edge聽auth#72 (same rules, /static exclusion).

~91% of ChapterController requests are 404s from automated scanners:
sensitive-file probes (SSH keys, cloud configs, .env files), Vite RSC
source-map probing, and other path guesses landing on the /:id chapter
catch-all. Return 404 at nginx instead of forwarding to Rails; every
blocked pattern already 404s in the app, so client-visible behaviour
is unchanged and the load moves off Rails.

Two regex locations in config/nginx.conf.erb:

- sensitive or unknown file extensions (nested paths included),
  excluding the real asset dirs and the advertised /sitemap.xml.gz
- extension-less probes seen in the canonical log archive (id_rsa-
  style key names, wp-* paths, /__vite_rsc_findSourceMapURL, the
  debug-trigger path)

Code review findings shaped the final rules: the extension rule
originally 404'd /sitemap.xml.gz, which public/robots.txt advertises
(fixed with an exact-filename exclusion so backup-archive probes like
site.tar.gz stay blocked), the .env branch missed .env.local and
.env.production variants (fixed), and the comment misstated nginx
location precedence (corrected: only exact = locations beat the regex
blocks).

New nginx CI workflow renders the ERB, runs nginx -t, and asserts a
curl matrix: scanner probes 404 at the edge, legitimate paths
(including /sitemap.xml.gz, chapter slugs, and the exact-match
Plausible proxy paths) pass through.

Verified against a live nginx:alpine container: the full probe and
passthrough matrix passes.

@olleolleolle olleolleolle left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enjoyable to have a CI test run for nginx config!

@mroderick
mroderick enabled auto-merge September 17, 2026 08:11
@mroderick
mroderick merged commit ade8fa6 into master Sep 17, 2026
10 checks passed
@mroderick
mroderick deleted the feature/scanner-junk-edge-block branch September 17, 2026 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants