Re-encode zero-width/format characters as entities in the qmd writer; accept them raw in prose

[cscheid]

Fixes #672.

Since the named-entity decode (#488) the reader folds ​ into a Str holding the literal U+200B, but the qmd writer emitted that codepoint verbatim and the grammar's prose regexes accepted no Cf character except ZWNJ/ZWJ, so pampa's own output failed to re-parse. Inside a heading superscript (quarto-web's # Welcome to Quarto^​[®]{.trademark}^) the failure cascaded into Q-2-16 Unclosed Superscript.

What changes

Writer (caa29d6c): escape_markdown spells every Unicode format character (general category Cf) as a character reference — the preferred WHATWG name where one exists (​, ­, ‎, ‏, ⁠, ⁡, ⁢, ⁣), else &#xXXXX;. ZWNJ/ZWJ stay raw: they already parse, they are ordinary content in Persian/Indic text, and ZWJ is structural inside emoji sequences. The Cf range table and the preferred names live in the new writers::format_chars, with a comment linking the WHATWG table and the alias rule, pinned by tests against the regex crate's \p{Cf} and against the shared entity table.

Grammar (caa29d6c): PANDOC_COMBINING_MARKS widens from \p{M}\u{200C}\u{200D} to \p{M}\p{Cf} (a strict superset), so raw zero-width characters — pasted from the web, or written by older pampa builds — parse as verbatim Str, matching Pandoc. Raw format characters are therefore accepted but not canonical at the text level: they re-emit as references. (Pandoc's readers behave the same way: ⁡, ⁡ and a raw U+2061 all become one Str, so the source spelling is not recoverable and the writer emits the preferred name for all three.)

& escaping (b15868ed, bd-i18zoy4n, folded in): the writer never escaped &, so a literal Str "©" (from \©) re-read as ©. With the writer now emitting references itself this became load-bearing. starts_character_reference mirrors the grammar's two tokens (numeric refs by regex shape; named refs via the reader's own entity_table()) and emits \& only for those — AT&T, a & b, &amp, &AM; are untouched. Pandoc's markdown writer escapes the same way.

Tests

All written and verified failing first: 7 corpus cases (test/corpus/format_characters.txt), 8 parser coverage tests, 9 writer unit tests, 4 pin tests, and four round-trip fixtures (named_entities.qmd extended; format_characters_raw.qmd, zero_width_in_heading_superscript.qmd, ampersand_escaped_entities.qmd new). No existing test or snapshot changed. parser.c regenerated (+496/−489 lines, character-class rows only; generate took 0.26 s).

Full cargo xtask verify (including the hub-client/WASM legs) green for both code commits.

End-to-end

$ printf 'a​b\n' | pampa -t qmd | pampa
[ Para [Str "a​b"] ]                       # was: Parse error

$ cargo run --bin q2 -- render gh672.qmd   # quarto-web heading + raw U+200B + \©
<h1 class="mt-1">Welcome to Quarto<sup>​<span class="trademark">®</span></sup></h1>
<p>Raw zero-width: a​b and soft hy­phen.</p>
<p>Literal reference kept as text: &amp;copy; and &amp;ZeroWidthSpace;.</p>

Follow-up

• bd-5rr4lgj1: a leading UTF-8 BOM (U+FEFF) is not stripped by the reader. After this PR a BOM-prefixed file parses (BOM inside the first Str) instead of erroring; Pandoc strips it, and so should we.

Plan: claude-notes/plans/2026-09-11-zero-width-chars-writer-parser-gh672.md. Braid: bd-wuiu1of7, bd-i18zoy4n.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EMPGiXmtk1yMv5YVVPPT9H

[posit-snyk-bot]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.