Add quarto call axe, a hidden experimental site accessibility scanner - #14815
Open
cwickham wants to merge 51 commits into
Open
Add quarto call axe, a hidden experimental site accessibility scanner#14815cwickham wants to merge 51 commits into
quarto call axe, a hidden experimental site accessibility scanner#14815cwickham wants to merge 51 commits into
Conversation
cwickham
added a commit
that referenced
this pull request
Aug 27, 2026
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
cderv
reviewed
Aug 31, 2026
cderv
left a comment
cderv
left a comment
Member
There was a problem hiding this comment.
First round of review for our 📲
This was referenced Aug 31, 2026
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
The cap was applied to the sorted path list before any file was read, so redirect stubs spent it. On a site whose first *.html is a stub, --max-pages 1 discovered nothing and the scan exited 2 with real pages sitting right there. Walk and classify together, stopping once the cap holds that many real pages. A capped scan's stub ledger now only covers what was walked before the cap filled — consistent with the rest of a subset scan, whose counts already describe the subset rather than the site. Reported by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
A flag typo produced the same signal as a real finding: axeScanConfig throws before axeScan's own handling, so it fell through to quarto's root handler, which exits 1 — the code this command reserves for 'new findings at the --fail-on threshold'. --themes matching nothing was wrong the other way, exiting 2 as if the scan hadn't finished. Both throw sites now raise the same named error and land on 3. A typo is not a result, so it doesn't borrow a code that means one. Reported by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
Two entries for one signature merged: union the pages, take the most severe accepted impact. Acceptance records the impact it was written at and re-alerts on escalation past it, so 'most severe' is the lenient direction — a minor acceptance beside a critical one became 'critical, everywhere', and the stricter page inherited the looser page's tolerance in scope and impact both. Not a copy-paste slip worth absorbing, either: axe assigns impact per check rather than per rule, and a finding's impact is the worst across the occurrences in that scan, so two honest entries can disagree. parseBaseline now reports each repeat, reading the raw data so a duplicate and a typo surface in the same pass. acceptances() builds the map straight and treats a duplicate as an internal error. Suggested by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
As a post-render script the command scanned whatever was in the output dir, including after 'quarto render index.qmd' — findings describing a mixture of this render's output and the last one's. Post-render scripts do run on incremental renders; the comment at project.ts:830 claims otherwise but the only guard there is 'if (!projResults.error)'. Skip with a note and exit 0, so an incremental render still succeeds. Preview reloads leave RENDER_ALL unset too, so they skip as well. The signal is an inference and says so in the comment: quarto has no 'am I a project script' flag, and QUARTO_PROJECT_OUTPUT_DIR is the closest thing — present for both script phases, absent otherwise. QUARTO_PROJECT_OUTPUT_FILES would be precise but goes missing under the QUARTO_USE_FILE_FOR_PROJECT_OUTPUT_FILES escape hatch. Suggested by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
Two holes in the one property the scanner is sold on — that a cell always finishes. The recovery after a timeout sent Page.navigate with no deadline. If the transport itself was what wedged, that send never returned and the scan stopped finishing cells: a hang, not a timeout. Both recovery commands now go through sendWithin, which gives up after a second and never rejects. Only the load-event wait was capped before. And a timed-out cell was abandoned, not stopped: its in-flight round-trip still completed, and its next step ran against the page the *following* cell had navigated to — a stale Runtime.evaluate injecting axe and starting a second scan underneath it. Sends from the run body now go through a wrapper that refuses once the cell is abandoned. Both are pinned by unit tests over a stubbed transport; without the fix the wedged-transport one hangs until the harness's two-minute guard. Reported by cderv in review of #14815.
cwickham
added a commit
that referenced
this pull request
Sep 3, 2026
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
Member
Author
|
Thanks for this — a good first round. Every thread has a reply. Seven
All seven passed roborev, and the axe suite is green: 162 unit tests and the Four of your ideas I have kept as follow-ups rather than folding into this One question back, in the |
cwickham
marked this pull request as ready for review
September 3, 2026 17:00
Member
Author
|
@cderv round one is fully addressed — every thread has a reply, and six of them have a commit behind it. Since those replies I also merged Ready for another look. If it looks good to you, I think it is ready to merge. Two things that follow this PR rather than block it:
One question is still open, in the |
Registers `axe <site-dir>` under the already-hidden `dev-call` parent, with the six v1 flags (--pages, --max-pages, --viewports, --themes, --timeout, --settle) parsed into a typed AxeScanConfig and echoed. The scan, aggregate and report stages follow.
Ports the quarto-web harness's raw-CDP driver to typed TypeScript and replaces its render-time axe hook with scan-time injection: after load and settle, the vendored axe.min.js (4.10.3) is evaluated in the page and axe.run() awaited under the per-cell timeout. Any rendered site now scans as-is, offline, on one known axe version — so the CDN warm-up hack, the console-payload sniffing and the special render all come out. The orchestrator serves the site dir with httpFileRequestHandler, launches headless Chrome via getBrowserExecutablePath() on its own throwaway profile, walks the page x viewport x theme matrix, and writes raw per-cell JSON to _axe-checks/cells/. Cells fail closed: timeout, evaluation error or a missing payload is reported and exits 2, never a pass. Chrome rewrites its profile as it shuts down, so the temp profile dir is removed only after the process has really exited. On Ctrl-C an onCleanup handler kills Chrome and leaves the dir behind, since cleanup handlers can't await. Verified against a rendered website: all cells ok; identical results with Chrome's DNS blackholed (offline); prefers-color-scheme emulation catches a dark-only color-contrast failure and the mobile viewport a mobile-only one; a cell that blocks the main thread times out and the next cell still scans; SIGINT leaves no orphan Chrome; bogus, empty and non-directory site-dirs exit 2. Adds an optional onListen to handleHttpRequests so the scan's own progress output isn't preceded by Deno's "Listening on ..." line.
Neither exclusion did any work here. Tabster's `[data-tabster-dummy]` sentinels arrive with the preview client's Fluent UI bundle, and `.quarto-axe-report` is the in-page panel axe-check.js builds when a site is rendered with `axe:` metadata — so neither appears in the plain rendered sites this command scans. Both were carried over from axe-check.js's own axe.run() call, along with a "won't fix upstream" claim that tabster#288 does not support: that issue is still open with no maintainer response. If someone does scan an `axe:`-rendered site, the overlay's own violations are information rather than noise. Results on the test site are unchanged: 12 cells, 12 ok, same rule ids.
Ports the harness's aggregate and report stages, with the semantics the notes had already settled but the harness code had not caught up with. Signatures use the investigated "option 1" normalization: attribute values are kept with digit runs wildcarded, rather than stripped. Stripping turned `div[data-bs-target=".callout-4-contents"]` into the generic `div[data-bs-target]`, which every Bootstrap collapse, modal, tab and dropdown also matches — so one accepted callout defect silently suppressed unrelated conformance failures site-wide. The baseline is now the hand-written projection ledger: `signature` x `pages` x `impact`, empty `pages` meaning site-wide, a listed `pages` fail-closed at finding level, and escalation past the accepted impact re-alerting. `--update-baseline` is not ported; stale entries are reported and pruned by hand. Conformance labels come from `axe-check.js`'s own `axeConformanceLevel`, `impactRank` and `standardRank` rather than a second implementation, so a finding reads the same whether it came from a scan or the in-page report. That module guards its self-init on `typeof document`, which is what makes it importable here — the same property tests/unit/axe-*.test.ts rely on. `schemas.ts` holds Zod for both contracts. Baseline validation needed more than a schema: to Zod, a misspelled `impcat:` is an ignored unknown key plus a missing `impact`, so the report blamed `impact` while the culprit sat two characters away — and a `superRefine` never runs once a required field is missing. `parseBaseline` runs both checks and merges the issues, so the error names the typo and suggests the field it meant. `_axe-checks/` and `_axe-baseline.json` now anchor at the project root (nearest `_quarto.yml` at or above the site dir), falling back to the working directory for loose HTML. They sit beside the output dir, never inside it: a full render of a website or book deletes the output dir. Verified end-to-end on a rendered site: findings.json satisfies its own schema; all five baseline behaviours (site-wide accept, page-scoped accept, page-scoped re-alert on an unlisted page, impact-escalation re-alert, stale entry) land as expected; two hand-typo'd fields produce named errors with suggestions and exit 2. The report was driven in headless Chrome — row drill-down, column re-sort with row/detail pairing intact, the why-accepted column, the stale notice, and the copy-AI-briefing payloads.
The signature is the most consequential value the scanner computes: too broad and one accepted defect suppresses unrelated conformance failures site-wide, too narrow and a baseline entry stops matching when a counter changes. Two gaps around it. First, nothing recorded which normalizer produced a signature. `version` covers findings.json's field shape, so a normalizer change would leave every field intact while re-keying every signature — reported as "N new findings, M baseline entries not seen", which is indistinguishable from "you fixed everything and broke an equal amount". The ledger's notes, the record of why each finding was accepted, would quietly stop applying. `signatureScheme` makes that a named error instead, and points at occurrences[].target as the unchanged raw selector to re-annotate from. It is optional in the baseline, so the first hand-written ledger needs no ceremony. Second, the normalization had no tests. tests/unit/axe-signature.test.ts covers it as explicit should-collapse / must-stay-distinct pairs over selectors axe really emits on Quarto output, rather than as assertions about the regexes. The must-stay-distinct half is the important one: it pins that a callout, a modal, a carousel and a tabset link do not share a signature just because they all hang off data-bs-target, which is exactly what the pre-option-1 normalizer got wrong. A third group asserts exact output so a scheme change shows up as a diff rather than as a changed collapse count. 30 tests, all passing. Also covers a case with no fixture yet: a color-contrast payload missing its check data must fall back to the selector rather than key on "undefined on undefined" and collapse every contrast finding in the site into one.
The two-pass reader was built on a wrong premise. I had it that a schema can't report an unrecognized key and a missing required field together, because a `superRefine` never runs once a required field is missing. That is true of `superRefine`, but strict-key checking is part of the object parse itself, so `.strict()` reports both in one pass: findings.0.impact: Required findings.0: Unrecognized key(s) in object: 'impcat' Adjacent lines make the typo obvious, so the hand-rolled Levenshtein "did you mean" was polish sitting on top of ~60 lines that Zod already covers. Deleted, along with the tolerated-field set and the manual per-entry loop; schemas.ts drops from 341 to 275 lines. Keeping the paste-a-whole-finding affordance costs one expression rather than the machinery: declare every finding field as an ignorable optional and merge the entry schema over the top, so its required fields stay required. A finding pasted out of findings.json plus a note validates; an invented field is still rejected. `parseBaseline` survives only for the signature-scheme check, which needs to explain what a mismatch means rather than say "invalid literal". Behaviour change worth noting: the outer object is `.strict()` too, so an unknown top-level key is now an error. That catches a misspelled `findings` but also rejects a hand-added `"_comment"`, since JSON has nowhere else to put one. tests/unit/axe-baseline-parse.test.ts pins the deliberate choices: the paste shortcut validates, a typo names both halves, an invented field is rejected, `pages` and `note` are required so scope and rationale are never guessed, and a stale scheme is one named error rather than mass staleness. 42 axe unit tests passing.
A purpose-built inaccessible Quarto site under tests/docs/axe-scan/site, with every violation planted on purpose and a manifest README recording intent. Source only: a committed render would carry site_libs/ and go stale, so the smoke test renders it in setup and removes it after. Writing the manifest first paid off twice, because two planted cases did not behave as designed. The tier-2 links were meant to prove that three `#panel-N` components collapse to one signature while a `#settings-dialog` stays separate. They all collapsed. axe had keyed them on `href`, which the normalizer drops as volatile, so the attribute never reached the selector. Switching to `<button>` — no href to key on — makes axe select on the attribute and the case works as intended. That distinction is the whole justification for keeping attribute values, so it is now proven on real markup rather than on hand-written selectors. The dark-only contrast paragraph also produced a second, unplanned contrast failure: it contained inline code, and cosmo's code colour fails against the planted background. That would have made the fixture depend on a theme palette. The paragraphs are now kept free of any element carrying its own colour, and the README says why. One finding is left unplanned on purpose: Quarto's navbar-toggler, which is systemic and mobile-only and belongs to Quarto. It is documented, and no assertion depends on it — or on any total count — so a Quarto upgrade that fixes it cannot fail these tests. Tests, 77 passing: - axe-aggregate.test.ts (19) drives the aggregate stage from eight verbatim per-cell captures. Verbatim matters: keeping whole payloads means an axe upgrade that changes their shape surfaces in a fast unit test. - axe-baseline-reconcile.test.ts (15) uses hand-built cells, where control beats fidelity: site-wide and page-scoped accepts, the re-alert on an unlisted page, escalation, de-escalation staying accepted, stale detection in three variants, and duplicate-entry merging. - smoke/axe/axe-scan.test.ts runs the real thing — browser, static server, all three stages — in 15s, asserting projections looked up by signature. The fixture's committed _axe-baseline.json exercises all four reconcile outcomes in one file, including an entry accepted at a lower impact than planted (so escalation re-alerts) and one matching nothing (so staleness is reported). Its `.a11y-accepted-example` entry is a designed future failure: when exclude-in-source ships, that entry should turn stale.
`quarto-bld validate-bundle` runs eslint's no-undef over the bundled
quarto.js, and it caught two things this branch introduced.
The first was importing `axe-check.js` to reuse its three conformance
labellers. That looked like good reuse — the labels match the render-time
`axe:` report by construction — but `axe-check.js` is a browser module, so
esbuild inlined the entire overlay (reveal navigation, dashboard rescan,
report DOM) into the CLI bundle for the sake of three pure functions, and
its page globals surfaced as four no-undef errors:
'Reveal' is not defined
'getComputedStyle' is not defined
'bootstrap' is not defined
They are `typeof`-guarded and harmless at runtime, but whitelisting them in
the prelude would blunt a check whose whole job is catching real bundler
bugs. conformance.ts now mirrors the three functions in TypeScript, and
tests/unit/axe-conformance-parity.test.ts imports both and asserts they
agree across every tag shape axe emits — best-practice, each WCAG
version/level, multi-digit and multiple criteria, obsolete criteria, and no
conformance tags. Tests aren't bundled, so that comparison is free. The
bundle lost ~400 lines and all four errors.
A shared pure module would be better than either, and is verified viable —
`axe-check.js` is injected with `type="module"`, and a FormatDependency's
`resources` land in the same output dir as its `scripts`, so both sides
could import one `axe-conformance.js`. Deferred: it changes the render path
every `axe:` user hits, and adds a runtime 404 risk, for a hidden
prototype's benefit. Recorded in the design note's cut list for the public
command.
The second was `TextDecoderStream`, used to drain Chrome's stderr. It is a
genuine Deno global simply missing from the bug-finder prelude's list, so it
is added there rather than worked around in the scanner.
After both: 1 error, pre-existing, in puppeteer code — the
`@typescript-eslint/ban-ts-comment` rule-not-found that the prelude's own
comment already documents. Also confirmed axe.min.js is *not* inlined: it
stays a runtime resource, so the bundle doesn't carry 553KB of axe-core.
No changelog entry. `dev-call` has never appeared in a changelog across
eleven versions, and an entry would advertise a hidden command that carries
no API commitment while its semantics are in play.
Headers audited (twelve new files, current year), deno fmt and lint clean,
113 axe tests passing, full typechecked run green.
The theme axis was measuring nothing on a default Quarto site. Emulating `prefers-color-scheme: dark` leaves the dark stylesheet `rel="disabled-stylesheet"` and the body white, because `respect-user-color-scheme` defaults to false — so every dark cell was an exact duplicate of its light sibling, and one of the two axes justifying the 2x2 default earned nothing. The axis tests the two themes the *author* ships. The user's OS preference is only one possible route into a theme, and a page may offer either route: Quarto's `light:`/`dark:` themes are selected by its colour-scheme toggle, author CSS under `@media (prefers-color-scheme: dark)` by emulation. Both are now set, pointing at the same theme, and the result is read back rather than assumed — which makes `respect-user-color-scheme` invisible here: when it is true, emulation has already done the work and no click happens. Alignment runs per cell, not once per run. The toggle writes `localStorage` and every cell shares one origin, so a click leaks into later page loads. This is not hypothetical: on the fixture, the final light cell now records `colorScheme: "toggled"` because it *loaded in dark* and had to be clicked back. Aligning once would have scanned it dark while labelling it light. Each cell records how it got there — `emulated`, `toggled`, `assumed-identical`, or `unreachable`. The last matters most: a page that ships a dark theme but renders no toggle cannot be switched, so its dark cell is really a light cell and is reported rather than counted as coverage. Cells on pages with no Quarto dark theme reuse their sibling's payload instead of running axe twice on the same DOM. That is an assumption, not a proof, and it is wrong for a custom `.scss` rolling its own dark mode or for bslib's web components, whose `prefers-color-scheme` rules live in shadow roots. A sound version — enumerate every stylesheet including shadow roots, fail closed on anything unreadable — was built and removed as unjustified complexity for v1; both design notes record the gap as an M5 question. Reuse never omits a cell, so the accounting stays honest, and `--themes dark` alone scans such a page properly. Evidence this works: the new `theme.qmd` plants a contrast failure in darkly's own palette (`#6c757d on #222222`, 3.39:1). It appears in dark cells only, and it was **invisible before this change** — proof distinct from `media.qmd`, whose dark-only failure comes from an author media query and was already reachable by emulation. The fixture now covers both routes separately. 81 axe tests passing. Results are unchanged by the reuse optimisation: 24/24 cells, 12 findings, both before and after.
Found by scanning a real site with a `_brand.yml` that specifies no dark values: the "reused a sibling result" note never appeared, though the site has no dark mode. A light-only `_brand.yml` makes Quarto emit the colour-scheme link machinery with `data-mode="light"` links but no dark build and no toggle. The probe deciding whether a theme cell can reuse its sibling asked for `link.quarto-color-scheme[data-mode]`, which matches those light links, so it concluded there was a dark theme to select and scanned both themes for nothing. The alignment probe next to it already used `[data-mode="dark"]` correctly; the loose selector came in when the capability probe was simplified. Effect was wasted work rather than wrong results — the dark cells scanned the same DOM and produced the same findings — but on a brand.yml site that is half the matrix, which is exactly the cost this reuse exists to avoid. The fixture had no such case, which is why nothing caught it. `static/brand-light-only.html` reproduces the markup by hand rather than adding a second fixture project: the probe runs per page, not per site, so one static page pins it. Verified by reintroducing the bug and watching the new assertion fail, then restoring the fix. Its planted violation is a bare `<button>`, not an alt-less image. An image normalizes to `img` and would join the systemic `image-alt` finding from `_systemic.qmd`, changing that finding's page list — which is how the first attempt failed, coupling a colour-scheme regression fixture to an unrelated grouping assertion. Fixture is now 7 pages, 28 cells, 13 findings. 81 axe tests passing.
One site and one smoke test shared a page list, so adding a page moved
assertions in unrelated verifiers — twice. Now sites/{findings,matrix,
brand-light-only} each carry one concern with their own smoke test, the
shared plumbing lives in tests/smoke/axe/shared.ts, and the captured
unit-test cells are grouped per site so each aggregates as a scan that
really happened.
Page discovery now reads each rendered page once: the inline before-body script marker (quartoToggleColorScheme) means a light/dark pair; its absence means one cell, labelled 'default'. The matrix is known and printed before the browser launches. Each cell selects its mode by seeding localStorage["quarto-color-scheme"] before navigation and verifying from the body class after load — which deletes AxeColorSchemeMechanism and all four states, sibling-payload reuse, alignColorScheme, the toggle click, the post-click settle and both console warnings, and no longer misses a light-coloured theme in the dark: slot the way the [data-mode="dark"] probe did. A one-mode page whose sole bootstrap link measures dark (theme: darkly) is annotated dark-coloured — in the cell payload and on the console, never in the cell's name. --themes becomes a filter over the discovered modes: it prunes the light/dark pair, 'default' cells are always included, and a filter matching zero cells is an error. findings.json gains a per-page modes field; report.html is unchanged. New fixture site sites/darkly covers the annotation; the research behind the design is in llm-docs/html-dark-mode-architecture.md, with its discovery pointer in .claude/rules/formats/html-dark-mode.md.
_axe-checks/ is the scanner's report from a previous run — when the anchor is the site dir (output-dir: "." projects, or loose HTML scanned from inside it) a rerun would scan its own report. site_libs/ is vendored library code: reveal ships plugin/notes/speaker-view.html with every deck, and its findings belong upstream, not to the site's author. Both are skipped at any path depth. Found playtesting (M5): braid br-b2-self-scan, br-skip-site-libs.
scanCell's run body had no try/catch, so a transport failure — a crashed tab, a dropped WebSocket, a CDP protocol error — propagated out of runAxeScan uncaught: process dead, no findings.json, no report, even with hundreds of good cells on disk. Now it returns an error cell; if the connection is really dead, later cells fail fast the same way and the exit code says incomplete, with everything scanned so far still reported. The orchestrator also catches what scanCell can't (shared setup, a stage bug) and reports incomplete instead of dying on an uncaught error. Braid: br-b1-transport-fail-closed.
Redirect stubs (a zero-delay meta refresh generated from _redirects) load locally, then Chrome follows the refresh during the settle window — axe was auditing the external destination and attributing its violations to the local page path (found on positron-website: opensource.posit.co findings under blog/index.html). Also broke the offline story silently. After settle, the cell probes window.location.href and fails closed with a new 'redirected' status — origin and path decide; search and hash are ignored so reveal's history-API fragment rewrites still count as the same page. Belt and braces: axe's own result.url is checked after the run, so a redirect between probe and run is caught too. Braid: br-redirect-escape.
The scanner writes _axe-checks/.gitignore ('*') on first run: a --pages
subset scan overwrites findings.json with a subset snapshot, so a committed
copy diffs as if findings were fixed. The committed contract stays
_axe-baseline.json, beside the directory.
cellName mapped / to _, so docs/index.html and docs_index.html shared an artifact name and the second write silently replaced the first; colliding slugs now get a short path-hash suffix, everything else keeps its name. Page paths are percent-encoded per segment before Page.navigate: a # or ? in a file name truncated the URL, scanning the wrong page and failing on the 404 with a misleading message.
A page merely mentioning quartoToggleColorScheme (docs about the toggle) false-triggered two-mode classification: the dark cell then failed closed with a misleading mode-mismatch error, or passed silently as a duplicate cell where no body class exists. Match window.quartoToggleColorScheme = instead; highlighted samples split the name across spans and no longer match. The emitting template carries a pointer back at the sniffer.
Deno.exit skipped onCleanup handlers; safe today (the browser is closed in finally) but it sidesteps the quarto convention.
--fail-on minor|moderate|serious|critical exits 1 when a COMPLETE scan has new (non-baselined) findings at or above the threshold — the CI regression signal. Exit 2 (incomplete scan) takes precedence: an incomplete scan never reads as a pass. Baselined findings never trip the threshold. The flag is echoed into findings.json's config (additive, no version bump) and into the README's regenerate command.
Deno.serve shuts down asynchronously after its abort signal fires; exiting mid-teardown intermittently segfaults Deno 2.7.14 (SIGSEGV null-call on a tokio-runtime-worker — three identical macOS crash reports, one predating this week's changes; 'segfault at exit' is a known upstream bug family). Await server.finished on every exit path to close the widest race.
config.ts parsing gets unit coverage through axeScanConfig (defaults, list trimming, viewport/theme/integer/--fail-on validation, AxeOptionError). Exit codes get end-to-end coverage in a subprocess — the in-process harness can't survive exitWithCleanup — over a hand-written static fixture: findings alone exit 0, --fail-on exits 1, a timed-out scan exits 2 even under --fail-on, with fail-closed artifacts left behind.
The committed architecture doc replaces pointers into private design notes: scan.ts, cmd.ts and aggregate.ts referenced 'the design note' and the signature-breadth investigation, which reviewers cannot read. It records the decisions that superseded those notes (markdown report over HTML, generated README as the agent enabler, extension as the rich report's eventual home). cri.ts cross-references the scanner's second Chrome launcher; a path-scoped rule surfaces the doc when the axe source is edited.
The 'how do I start' tier for early adopters: when to reach for the scanner vs the render-time axe: option, flags, the baseline workflow end to end, exit codes, and the CI recipe (--fail-on gate + report.md as a PR comment, with the 65,536-char comment cap noted). Contributor-facing on purpose — it graduates to quarto.org when the command sheds dev-call.
Pandoc parses the text between <code> tags as markdown, so an excerpt from a page that documents fenced divs put literal ::: strings into the AST and Quarto's fenced-div check warned on every render of the report (found playtesting on quarto-web). Backtick spans parse as Code inlines, which the check ignores; fences widen past embedded backtick runs, and pipes inside table cells escape as \| — the one escape both GitHub and Pandoc honor at the cell boundary.
cderv's suggestion: axe is a user-facing subsystem function, which is call's namespace — dev-call is for internal dev tools. The module moves wholesale (same depth, imports unchanged) and stays .hidden() while experimental: invocable, not advertised in quarto call help. Every surface that said 'dev-call axe' — the generated README's regenerate command, help text, tests, fixture READMEs, docs, changelog — now says 'call axe'.
QUARTO_CHROMIUM pointing at a file that exists but is not a browser wins discovery and then fails to spawn — the closest forcible stand-in for 'no usable Chrome' on machines that have one. Pins exit 2, the 'Could not start headless Chrome' message, and that a pre-existing findings.json does not survive to read as current. (Discovery finding nothing at all is core/puppeteer.ts behavior, shared with mermaid, and not forcible here.)
core's 'Chrome not found' throws an empty Error after printing its own explanation and the chrome-headless-shell install suggestion; the wrapper line now ends cleanly instead of 'Could not start headless Chrome: '.
The how-to claimed a shared-include defect is one finding 'however many pages repeat it'. That holds when axe emits selectors that normalize identically (fixture-verified: the shared include yields bare img on both pages) and can break for anonymous elements whose surrounding DOM differs — axe then picks different minimal selectors and one cause splits. Say so.
Verified live: 'quarto call axe _site/' as a project post-render script scans after outputs are written, from the project dir (trailing slash fine); without --fail-on findings never fail the render, with it a threshold trip or incomplete scan fails quarto render with the scan's error line in the output. CI recipe now uses marocchino/sticky-pull-request-comment so each push updates one comment instead of stacking new ones.
The --help output is the first thing anyone who discovers the hidden command reads; align it with the docs' no-stability wording and point at dev-docs/axe-scan.md.
quarto preview of the report fails inside a project (_-prefixed dirs are not project inputs; verified: 'No output created by quarto render report.md'), while a plain quarto render works standalone. Document the hierarchy: editor/GitHub preview first, render-then-open for HTML, --report into the site source for the previewable-with-site flow. Deliberately no _quarto.yml inside _axe-checks: it would make renders there pick up the README and adds magic to a disposable directory.
320 CSS px is the one viewport width WCAG names (SC 1.4.10 Reflow — the 400%-zoom equivalent of a 1280 window). axe has no automated reflow rule, but every rule runs against the reflowed layout, and 320 sits in the same Bootstrap breakpoint regime as any phone width, so the mobile chrome still renders. 390x844 was the iPhone 12-14 logical size inherited, undocumented, from the quarto-web harness. Signatures never embed viewports, so committed baselines keep matching; only viewport-gated findings can churn.
'The scanner needs six CDP methods' conflated protocol commands (scan logic, always ours) with CdpClient's capabilities (generic transport: send/once/close). Spell out the three layers — launcher, transport, scan logic — and which part of cri.ts was declined (its mermaid-shaped facade) vs what a future unification would actually share (the deno-cri transport underneath, and sooner, the launcher).
cderv read 'posting it as a PR comment is workflow configuration, not tooling' as possibly announcing a gap Quarto should fill, and two sentences ending in colons back to back did not help.
The cap was applied to the sorted path list before any file was read, so redirect stubs spent it. On a site whose first *.html is a stub, --max-pages 1 discovered nothing and the scan exited 2 with real pages sitting right there. Walk and classify together, stopping once the cap holds that many real pages. A capped scan's stub ledger now only covers what was walked before the cap filled — consistent with the rest of a subset scan, whose counts already describe the subset rather than the site. Reported by cderv in review of #14815.
A flag typo produced the same signal as a real finding: axeScanConfig throws before axeScan's own handling, so it fell through to quarto's root handler, which exits 1 — the code this command reserves for 'new findings at the --fail-on threshold'. --themes matching nothing was wrong the other way, exiting 2 as if the scan hadn't finished. Both throw sites now raise the same named error and land on 3. A typo is not a result, so it doesn't borrow a code that means one. Reported by cderv in review of #14815.
Two entries for one signature merged: union the pages, take the most severe accepted impact. Acceptance records the impact it was written at and re-alerts on escalation past it, so 'most severe' is the lenient direction — a minor acceptance beside a critical one became 'critical, everywhere', and the stricter page inherited the looser page's tolerance in scope and impact both. Not a copy-paste slip worth absorbing, either: axe assigns impact per check rather than per rule, and a finding's impact is the worst across the occurrences in that scan, so two honest entries can disagree. parseBaseline now reports each repeat, reading the raw data so a duplicate and a typo surface in the same pass. acceptances() builds the map straight and treats a duplicate as an internal error. Suggested by cderv in review of #14815.
As a post-render script the command scanned whatever was in the output dir, including after 'quarto render index.qmd' — findings describing a mixture of this render's output and the last one's. Post-render scripts do run on incremental renders; the comment at project.ts:830 claims otherwise but the only guard there is 'if (!projResults.error)'. Skip with a note and exit 0, so an incremental render still succeeds. Preview reloads leave RENDER_ALL unset too, so they skip as well. The signal is an inference and says so in the comment: quarto has no 'am I a project script' flag, and QUARTO_PROJECT_OUTPUT_DIR is the closest thing — present for both script phases, absent otherwise. QUARTO_PROJECT_OUTPUT_FILES would be precise but goes missing under the QUARTO_USE_FILE_FOR_PROJECT_OUTPUT_FILES escape hatch. Suggested by cderv in review of #14815.
Two holes in the one property the scanner is sold on — that a cell always finishes. The recovery after a timeout sent Page.navigate with no deadline. If the transport itself was what wedged, that send never returned and the scan stopped finishing cells: a hang, not a timeout. Both recovery commands now go through sendWithin, which gives up after a second and never rejects. Only the load-event wait was capped before. And a timed-out cell was abandoned, not stopped: its in-flight round-trip still completed, and its next step ran against the page the *following* cell had navigated to — a stale Runtime.evaluate injecting axe and starting a second scan underneath it. Sends from the run body now go through a wrapper that refuses once the cell is abandoned. Both are pinned by unit tests over a stubbed transport; without the fix the wedged-transport one hangs until the harness's two-minute guard. Reported by cderv in review of #14815.
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
cderv
pushed a commit
that referenced
this pull request
Sep 8, 2026
A bare repo path is no use to a reader of the changelog, so it is now a GitHub URL. The help text loses the pointer rather than carrying a long URL in a terminal: the quarto-web page that replaces dev-docs will get one, following the 'For details, see:' shape quarto run uses. Reported by cderv in review of #14815.
cderv
force-pushed
the
feat/axe-scan-command
branch
from
September 8, 2026 09:22
85c93b8 to
08742c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds
quarto call axe <site-dir>— a hidden, experimentalcommand that scans an already-rendered Quarto site for accessibility
violations with quarto-cli's vendored axe-core.
What it does:
as other Quarto features:
quarto install chrome-headless-shell, or asystem Chrome/Edge) over a page × viewport × color-mode matrix. Each
page's modes are discovered from its rendered HTML, so a light-only site
scans once per page and a
light:/dark:site scans both slots.as-is, offline, on the same axe version the render-time
axe:option uses.the color pair for
color-contrast), so one template defect on 40 pages isone finding with a count, and its id is stable across runs.
_axe-baseline.json):new findings alert, accepted ones are counted but out of the way,
escalations and out-of-scope recurrences re-alert.
_axe-checks/findings.json(machine contract),report.md(GitHub-flavored markdown), and a generated
README.mdthat documents theartifacts for whoever — or whatever agent — finds them in a repo.
infrastructure failure in the output and the exit code, never a pass.
Exit codes:
0complete,1new findings at/above--fail-on <impact>(opt-in, for CI),
2incomplete — and 2 takes precedence over 1.Why hidden under
call: the semantics (signature scheme, baselinegrain, flag surface) should be validated by real use before becoming public command. Nothing here is a public API commitment.
Docs in this PR:
dev-docs/axe-scan.md— how to use it (flags, baseline workflow, CIrecipe). Will eventually seed the
quarto-webdocs.llm-docs/axe-scan-architecture.md— how it works and why (pipeline, modediscovery, signatures, baseline semantics, the raw-CDP decision).
Deliberately not in scope (deferred to the public command):
_axe.ymlconfig, interaction states, ruleset scoping, source mapping (
.qmd→output), a rich report extension. The render-time
axe:option isuntouched; the scanner mirrors its conformance labellers, pinned by a parity
test.
Testing: 9 unit test files (115 tests) run the aggregate/baseline/discovery/
config logic browser-free against captured real axe payloads; 5 smoke test
files run the full command over fixture sites in
tests/docs/axe-scan/,including subprocess exit-code tests covering
--fail-on, fail-closedtimeouts, and a browser that cannot start. CI already installs
chrome-headless-shellunconditionally; a manualtest-smokes-paralleldispatch on this branch is green:
https://github.com/quarto-dev/quarto-cli/actions/runs/33106781443.
Checklist
I have (if applicable):
(not applicable yet: the command is hidden; docs live in
dev-docs/and graduate to quarto-web when it goes public)
AI-assisted PR