Skip to content

PT-4197: Data-fidelity audit — composed round trip, USX-first net, corpus - #29

Merged
tjcouch-sil merged 182 commits into
mainfrom
pt-4197-data-fidelity
Nov 3, 2026
Merged

tjcouch-sil merged 182 commits into
mainfrom
pt-4197-data-fidelity

Conversation

@tjcouch-sil

@tjcouch-sil tjcouch-sil commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the scripture-editors half of the PT-4197 end-to-end data-fidelity audit. It covers two things:

  • Test coverage: the full load and save trip over the real per-chapter USX from paranext-core's data provider, in every view a user can edit Scripture in.
  • Fixes: the Scripture losses and unasked edits that this work found. Where Paratext 9's behavior is the reference, the fixes match it.

Companion: paranext/paranext-core#2922, which holds the authoritative loss catalog .context/standards/Scripture-Data-Fidelity-Catalog.md. Merge this first, then move platform-yalc. Core's CI stays red until then: core has dropped its version shims and consumes the new editor option.

Why review this

Part of the Standard-view epic (PT-4186); PT-4197 is its data-fidelity work item. The goal is no Scripture data loss and no automatic edits the user didn't ask for.

Test nets

  • Composed round trip (corpus-pdp-usx-composed-round-trip.test.tsx): real data-provider USX → converters → mounted Editor → USX, in Standard, formatted and paragraph-structure (Simple mode's view).
  • USX-first converter net.
  • USJ-first converter net (testusfm-usj-usx-round-trip.test.ts) now discovers every oracle in the corpus, so 2SA-4, 2SA-5, 1SA-1 and spacing-word-break are covered too.
  • Unclosed spans under editing (unclosedSpanEditing.test.tsx): typing, Backspace and a paragraph split inside a span the USFM never closed, and typing before it, in every editing view; every span still saves closed="false" in the USJ and in the USX the provider receives. What ParatextData then writes from the editor's USX is pinned in paranext-core#2922's editor-first round trip.
  • One view registry (editorViews.test-helpers.tsx). Every applicable suite runs over the editing views from it.
    • Each view is labelled with the app surface it guards (power-standard, simple, footnotes-pane-row, preview, read-only-panel, library-only). The footnotes pane's row editor is a named entry (footnotes-pane-row), and READ_ONLY_VIEWS lists Power mode's Preview and the text grid's block-verse column, mounted read-only.
    • Read-only sweep (readOnlyGestures.test.tsx, 110 tests): every gesture (typing, Enter, deletes, paste in three flavors, cut, the shortcuts, drop, undo/redo, and the ref's editing methods) leaves every read-only view unchanged, and each gesture is shown to change an editable view.
    • Library-only views are opt-in. Views Platform.Bible doesn't offer (editable formatted, unformatted, the Markers view, Standard with expanded notes, the popover over a non-Standard host) run only with SE_TEST_LIBRARY_VIEWS=1 (pnpm test:library-views in packages/platform); CI and the default run skip them. That now includes the ~95 tests that built Standard-with-expanded-notes options themselves (mountExpandedNoteEditor, testEnvironmentExpanded, or the options spelled out) rather than naming the registry view. Default platform run ~339 s, ~587 s with them (at 8163d705).
    • Rows that tested app-reachable code only through a library-only view now have twins in the app's views: read-only suites run Preview instead of formatted, decorator rows run in Preview and Simple, note-content clipboard rows run in the footnotes pane's row editor, plus new rows for HTML paragraphs pasted in Simple and a pending \cat edit in the footnotes pane.
    • Of those ~95 Standard-with-expanded-notes tests: 9 were already repeated by the Standard rows of the same tables; 34 got twins in the footnotes pane's row editor over the one-note scaffold it mounts (paneRowScaffold / mountInPaneRow): note-content annotations, settles, whitespace, separators, the \cat run, caller-end and closer locations, adaptor rows; 45 run in Standard view instead, because Standard shows a note with no closer expanded with its shell editable, or the row never needed expanded notes (husk and stale-basis positions, selectNote, the paragraph-prefix default); 5 stay library-only (a closed note's closer as editable glyph text, and content pending inside a collapsed note). Each new row was turned red by breaking the behaviour it pins. Flag-off coverage keeps every production statement (lines/statements/branches 19,151/21,930/15,970 → 19,150/21,930/15,975); the only lost branches are the closer half of a closed note's opener rename, which needs an editable closed-note shell.
  • Typing-fidelity oracle for formatted and paragraph-structure. For every keystroke at every caret position it checks:
    • only the edited spot changed;
    • nothing was invented or lost;
    • Undo and Redo are exact.
  • Copy/cut/paste corpus: 543 rows.
  • Per-keystroke content check in the pending/settled oracle.
  • Corpus: testUSFM 2SA-4, 2SA-5 (damaged but at rest) and 1SA-1 (introduction), plus the pdp-usx/ captures.

Tickets filed by this work: PT-4909, PT-4910, PT-4911, PT-4921, PT-4924, PT-4925, PT-4931 (long-term real-NBSP representation) and PT-4952 (Standard view's character-marker remove/replace). The deferred delta-op format items (span identity in ops among them) went to the existing PT-4399.

Known defects are pinned as shrink-only expected failures, each naming its ticket. For example, the clipboard pins name PT-4911 and PT-4921, and the typing pins name PT-4924. When a fix lands, the pinned test starts passing and fails the suite.

Fixes

  • USX/USJ version is preserved end to end. Usj.version is now string. A declared version that isn't 3.x passes through unchanged too; the converters don't validate or upgrade it (TSDoc, 2.5 test cases).
  • PT-4391: an authored lone NBSP survives outside Standard view, including the cases where editing leaves it alone in a span.
  • Projects with Paratext's AllowInvisibleChars on: the new documented option ViewOptions.allowInvisibleCharacters keeps ~ and U+00A0 verbatim. Standard view used to turn a literal ~ into an NBSP on save.
    • The editor holds a no-break space as a U+001A stand-in, drawn as a blank by a one-glyph webfont. Every boundary passes a real U+00A0: change ops, note ops, the footnote popover, applyUpdate, the Lexical copy flavor, async paste, drag/drop, annotation text and the caller preview (nbspStandIn.test.tsx). Incoming C0 control characters become spaces in Standard view.
    • An attribute value keeps its ~ and its U+00A0 apart under either setting, as Paratext 9 does (catalog O-12; attributeValueNbsp.test.tsx, usfmFragmentToUsj.test.ts). Paratext 9 reads ~ as a no-break space only in body text. Standard and Unformatted views now hold a value's U+00A0 as the stand-in with the setting on or off, only the stand-in reads back as U+00A0, and a value's ~ is never touched; with the setting off a settle used to save the author's ~ as U+00A0. This covers milestones, char-span attributes, figures, \cat, \ca/\cp, \va and a folded \vp; \periph, an unfolded \vp and the bytes an unwrapped span leaves behind stay body text.
    • A U+00A0 pasted with the caret inside a value saves as U+00A0 with the setting on and as ~ with it off, as Paratext 9 pastes one; it used to save as a space. A value's U+00A0 copies as U+00A0 under either setting, and toUsfmNbspSpelling always turns the stand-in into U+00A0.
    • The stand-in's font family is no longer exported (NBSP_STAND_IN_FONT_FAMILY is gone; hosts name PlatformEditorNbspStandIn), and toUsfmNbspSpelling is marked @experimental, since both go away once Standard view holds a real no-break space (PT-4931).
  • Paste in Standard view:
    • a pasted no-break space at the start of a line, or on its own, is kept as a no-break space (pastedNbsp.test.tsx);
    • a pasted U+00A0 is read as a marker's separator only right after an opening marker, so a stored no-break space beside a closer or before an opener survives the editor's own copy and paste.
  • Delta ops:
    • Standard view's ops carry stored whitespace the way every other view's do, under both settings. This fixes losses in Simple mode's footnotes pane, which edits a paragraph-structure editor's notes in Standard view (noteOpsWhitespace.test.tsx);
    • an author's lone no-break space is kept in op content, and one applied as a char span's whole text is marked authored (appliedNbspContent.test.tsx);
    • text carried out past a note keeps its char spans and milestones (noteCarriedText.test.tsx);
    • milestone ops are emitted under ms, the key the op types declare; applyUpdate still accepts the old milestone key;
    • adjacent char spans whose attributes differ are no longer merged when the footnote editor loads a note from its ops (a second \w span kept its lemma), and spans brought in by one paste stay separate (adjacentCharSpans.test.tsx);
    • in the unformatted view, a note's ~-only span (\ft ~\ft*) and a ~ before a nested span survive the note's ops, the footnote popover and the footnotes pane, and delta-doc positions count an authored \nd ~\nd*, so later positions match the ops stream (lone-nbsp-agreement.test.tsx; agreement tests adapted from Jolie Rabideau's PT-4391: Keep a lone ~ between markers on save in every view #36).
    • an applied insert that extends an existing char span no longer pushes every later op in the same delta too far along, and an applied empty char span gets one placeholder no-break space, not two (delta-apply-update.utils.test.tsx);
    • getOpsAfterNote/takeOpsAfterNote report an empty char span after a note once, so text the footnote editor carries out past a note no longer builds two spans (noteEditorCloseCarriesRest.test.tsx);
    • a char span's separator stays U+00A0 in the ops the editor sends after a retain: an undo that restores it no longer sends the host a leading content space (separatorOpsAfterUndo.test.tsx). Incoming ops are spelled without the document's text, since $applyUpdate places them in apply positions, so a host's no-break space inside a span stays one (misalignedNbspApply.test.tsx);
    • an applied update that removes the paragraph after a \b without bringing any of its content in leaves the blank line's marker alone; a join still keeps the second paragraph's marker, and a range from the blank line over several paragraphs keeps the marker of the paragraph it ends in (ParaNodePlugin.test.tsx, typingFidelity.regressions.test.tsx);
    • $getApplyLength counts each leaf with the shared $getNodeOTContribution(node, "apply") rules.
  • Invisible characters are typed, pasted and copied as in Paratext 9 (TJ's rulings, 2026-10-29; InvisibleCharactersPlugin, invisibleCharacters.test.tsx):
    • with allowInvisibleCharacters off, a no-break space typed as a character goes in as a space in every view, not only Standard (Simple saved ~); a pasted one is still a no-break space, and with the setting on both are kept;
    • a zero-width no-break space (U+FEFF) typed, pasted (every flavor) or dropped is dropped, in every view and either setting; one the document already holds is kept by the editor, so loading or editing elsewhere never changes it (paranext-core#2922 drops it from what an edit saves, as Paratext 9 does);
    • with the setting off, a view that hides markers copies a stored no-break space as ~ in its plain text; layout spacing (gutter markers, empty-span placeholders, note spacers) and the html and internal flavors keep U+00A0, which the view's own paste reads back as the no-break space.
  • A read-only editor no longer takes a paste from the keyboard. ClipboardPlugin listens on the editor root, outside Lexical's read-only gate, so Ctrl+V / Ctrl+Shift+V pasted into the Preview, resource panels and the text grid. The shortcuts now paste and cut only in an editable editor; Ctrl+C still copies, and Ctrl+X no longer dispatches a cut there.
  • Copying a whole note through the editor's own copy is pinned (wholeNoteCopy.test.tsx).
  • Footnotes:
    • An empty note caller is read and written as caller="", as ParatextData does; an every-view test locks it (O-4).
    • Enter inside an expanded note does nothing where markers aren't editable (NoteEnterGuardPlugin); \fp support is PT-4909.
    • A nameless character marker in a note keeps its empty name.
  • Typing in hidden-marker views:
    • range edits across nested spans and footnotes;
    • joining into an empty \b keeps the joined paragraph, as Paratext 9 does; deleting an empty paragraph after a \b (Delete or Backspace from inside it) leaves the blank line a \b;
    • Delete before a collapsed note no longer crashes;
    • Undo of Enter;
    • Backspace between a span and a collapsed note;
    • alike spans stay apart, as in Standard view, while a span that Enter split is one span again when Backspace joins the paragraphs back;
    • Enter at a word break drops the space in Simple, as in the other views;
    • Undo after Simple's two-press removal of an empty paragraph (an empty \b) brings it back;
    • Redo works after an Undo leaves the caret beside an empty verse or a trailing note (the invisible caret host is no longer an undo step of its own).
  • Applying a character marker in hidden-marker views:
    • a newly marked span stays apart from an alike span beside it (marking "God" after \nd Lord\nd* makes a second span), as Paratext 9 and Standard view do;
    • a selection overlapping part of an alike span extends that span: a deliberate departure from Paratext 9, which mangles the line there (TJ's ruling);
    • applying a marker over a selection that starts or ends inside a span no longer hangs the editor (adjacentCharSpans.test.tsx).
    • styling a selection of exactly one stored ~ keeps it inside the new span; it saved an empty span and lost the ~ (lone-nbsp-gestures.test.tsx).
  • Applying a character marker across a span's marker border (markerApplyAcrossSpans.test.tsx):
    • Standard view refuses the apply, logging a warning and changing nothing, when the selection's two ends sit in different spans (an end inside a closing glyph counts as crossing) or inside one glyph; this fixes the \fq endless-transform loop;
    • in every view, a selection holding a whole span or note moves it into the new span intact, which fixes a freeze, a lost \nd and a note's content being marked instead of the note;
    • the leading space of a selection stays outside the new span;
    • the remaining cross-border behaviours are tracked on PT-4609.
  • Removing or replacing a character marker over a whole span in hidden-marker views (charMarkerSelectionEdge.test.tsx): a double-click or drag over a whole span ends at the start of the text after it, which Lexical records as a point on the span element. $getSelectionOffsets now reads that as the end of the span's text, so removing \add from "God" no longer leaves G\add od\add*, replacing it no longer splits it, and removing \nd from a span ending in a nested span works. The same fix covers extend and the apply path's same-marker check. Pre-existing on main. Standard view's remove/replace is broken even for plain selections but is not reachable from the UI; tracked on PT-4952.
  • In Simple mode, a Backspace or Delete that would run into a chapter is refused, including word and line deletes and deletes with no keydown. The letters beside a chapter stay editable (2a67a7e3), and an empty paragraph beside a chapter can still be removed. A range selected across a chapter is not guarded: one press removes it, chapter included, and the two paragraphs join.
  • Structure protection holds for edits that arrive with no keydown in the views that hide markers: under protected, an input method's composition, text insertion or a character delete (macOS Ctrl+H) over a range holding a verse marker or crossing a paragraph is refused, and under guarded such a delete arms the two-step confirm as Backspace does (inlineRangeStructureProtection.test.tsx).
  • Typing over a range whose two edge texts are wholly selected keeps the text after a char span outside it.
  • An input method or dead key composing over a range across a span edge or a collapsed note now takes only the selected text, as typing does (InlineRangeEditPlugin): \wj Lord said\wj* to them with "said to" selected composes to \wj Lord é\wj* them, where the following text used to land in the span. The plugin handles the placeholder Lexical inserts when a composition starts, so the selected text goes at the same moment it always did.
  • Typing only a zero-width no-break space over a selection leaves the selection in place. The character is dropped; it used to delete the selection.
  • Enter just before a space followed by a no-break, ideographic or zero-width no-break space no longer removes that character with the space; Paratext 9 keeps it after a marker.
  • Clipboard:
    • pasted text stays inside the span or note it was pasted into;
    • a paste over a range across a char span's edge replaces only the selected text in the views that hide markers, as typing does (InlineRangeEditPlugin, pasteOverInlineRange.test.tsx). It used to pull the text after the range into the span, and with nested spans put it ahead of the text it followed. A range that starts where prose meets a span starts in the span, so the pasted text goes in it, again as typing does;
    • a cut no longer strands an NBSP.
  • Paratext 9 parity in Standard view:
    • a char span's leading spaces, and a spaces-only span, save as ParatextData saves them;
    • the pending read shows the space before a verse.

AI Involvement

AI-assisted — Claude Code sessions ccea61eb-07b0-4188-96a6-86deef7a1698, this session and two code-review sessions (1, 2), and review-response sessions for the 2026-11-02 reviews (1, 196b7ee4-932a-4fb3-826a-3f23c824acab). Claude subagents generated the plans, tests, fixtures, fixes and triage under a controller session, with independent reviews of the riskier fixes. Claims about Paratext 9 were measured by running ParatextData. TJ directed scope and made every verdict decision.

Testing

At 7bf282e4 (hand-QA fix: usxStringToUsj drops an empty vid, so markers typed into a paragraph that continues a verse settle in Standard view; dist rebuilt and verified current):

  • nx run-many -t typecheck lint clean; packages/platform 5290 passed with six 2SA-5 clipboard pins (attributed to PT-4388) now passing and removed; the rest of the PT-4388 pin narrowed to the rows that copy the nameless paragraph marker; utilities, shared, shared-react passed. The new tests were red with the old converter.
  • paranext-core restaged on this tip: platform-scripture-editor 5753, platform-scripture 975 and platform-bible-utils 774 passed; the editor-usx resave captures lost only vid="", and ParatextData writes the same bytes from them.

Earlier, at 84465cb0 (the fourth review's three fixes: a protected HTML paste goes through the clipboard-nodes command, a pasted no-break space that Enter strands alone in a span is kept, an over-limit copy writes stored no-break spaces as ~; dist rebuilt and verified current):

  • nx run-many -t typecheck lint clean; packages/platform 5288 passed, 63 expected failures, 142 skipped; utilities, shared, shared-react passed. Each new test was red before its fix.
  • paranext-core restaged on this tip: platform-scripture-editor 5753 and platform-scripture 975 passed; the editor-usx captures regenerate unchanged.

Earlier, at 6e2473a1 (latest main merged: #45 context menu closes on window blur; dist rebuilt and verified current):

  • nx run-many -t typecheck lint clean; packages/platform 5254 passed, 63 expected failures, 142 skipped; utilities, shared, shared-react passed.

Earlier, at cf5d499d (test-only: the USJ-first net over every oracle, the unclosed-span editing suite):

  • nx run-many -t typecheck lint clean for all 10 projects.
  • packages/platform 5254 passed, 63 expected failures, 142 skipped; utilities, shared and shared-react passed; unclosedSpanEditing.test.tsx also passes with SE_TEST_LIBRARY_VIEWS=1 (30 tests).
  • No dist change on this tip (committed dist identical to e0e96429).

Earlier, at e0e96429 (the third review: structure protection for no-keydown edits, the \b empty-paragraph delete, version pass-through docs; dist rebuilt):

  • packages/platform with SE_TEST_LIBRARY_VIEWS=1 6900 passed, 70 expected failures, 28 skipped; the whole repo with SE_TEST_LIBRARY_VIEWS=1 (one commit earlier, before the attribute-paste revert) 10328 passed, 70 expected failures, 29 skipped.
  • nx run-many -t typecheck -p platform shared-react utilities and nx lint platform clean (existing warnings only); verify-committed-dist.mjs ✓, run last; index.d.ts is the full rollup.
  • paranext-core was not restaged against this tip, and the e2e specs were not rerun for this round.

Earlier, at 314da06e (the second 2026-11-02 review: incoming separator-ops revert, multi-paragraph \b join, $getApplyLength; dist rebuilt):

  • packages/platform default run 5228 passed, 63 expected failures, 142 skipped; libs/shared-react 2373 passed, 1 skipped.
  • tsc --noEmit (lib configs) and eslint clean in shared-react and platform (existing warnings only); verify-committed-dist.mjs ✓, run last.
  • The SE_TEST_LIBRARY_VIEWS=1 run and the e2e specs were not rerun for this round.

Earlier, at 45c6bcd6 (the 2026-11-02 review fixes, dist rebuilt; main has nothing newer):

  • packages/platform with SE_TEST_LIBRARY_VIEWS=1 6880 passed, 70 expected failures, 28 skipped; the default run, last at 796c589f (before the separator-ops fix), 5209 passed, 63 expected failures, 142 skipped; libs/shared-react 2373 passed, 1 skipped; libs/shared 946.
  • nx run-many -t typecheck lint -p shared shared-react platform 0 errors (existing warnings only); the dist rebuild was the last build, index.d.ts the full rollup, no stray Editorial.d.ts.
  • paranext-core was not restaged against this tip, and the e2e specs were not rerun for this round.
  • TJ needs to finish hand QA.

Earlier, at bea1c90f (the paste and typed-rewrite fixes, main merged including #38 and #39, dist rebuilt), with paranext-core 219408be716 consuming the editor staged from it:

  • packages/platform default run 5207 passed, 63 expected failures, 142 skipped; with SE_TEST_LIBRARY_VIEWS=1 6861 passed, 70 expected failures, 28 skipped; libs/shared-react 2367 passed, 1 skipped; libs/shared 945; packages/utilities 89.
  • nx run-many -t lint typecheck --projects=platform-editor,shared-react 0 errors (existing warnings only); verify-committed-dist.mjs ✓ after the dist commit, no stray Editorial.d.ts, index.d.ts the full rollup.
  • paranext-core: typecheck and lint clean; core 7479 passed, 1 skipped; extensions 7528; dotnet 2964 passed, 14 skipped; all nine no-unasked-edits e2e specs passed on the first attempt (details in paranext-core#2922).

Earlier, at 7912550a (attribute-value ~ fix, value paste, leading-space trim, composition over a range, review follow-ups, dist rebuilt), with paranext-core f2fd54eb9ed consuming the editor staged from it:

  • packages/platform default run 5035 passed, 63 expected failures, 142 skipped; with SE_TEST_LIBRARY_VIEWS=1 passed; libs/shared-react 2331 passed, 1 skipped; libs/shared 934; packages/utilities 89 (shared and utilities were last run one source commit earlier, before the composition fix, which touches neither).
  • nx run-many -t typecheck lint over all 10 projects: 0 errors (existing warnings only); verify-committed-dist.mjs ✓ after the dist commit.
  • paranext-core: typecheck and lint clean; core 7470 passed; platform-scripture-editor 5544; dotnet 2964 passed, 14 skipped; all nine no-unasked-edits e2e specs passed (details in paranext-core#2922).
  • TJ checked by hand in the Windows app: Japanese input and the Spanish dead key at a caret, and composing over a selection across a span edge.

Earlier, at 04e3a9df (the Paratext 9 whitespace rulings, CHANGELOG entry, dist rebuilt), with paranext-core beb55bb6fdd consuming the editor staged from it:

  • packages/platform default run 5024 passed, 64 expected failures, 142 skipped; with SE_TEST_LIBRARY_VIEWS=1 6551 passed, 71 expected failures, 28 skipped; libs/shared 920; libs/shared-react 2328 passed, 1 skipped; packages/utilities 89.

  • nx run-many -t lint typecheck -p shared shared-react platform-editor utilities 0 errors (existing warnings only); verify-committed-dist.mjs ✓ after the dist commit, no stray Editorial.d.ts, index.d.ts the full rollup.

  • paranext-core: typecheck and lint clean; core 7470 passed, 1 skipped; platform-scripture-editor 5530; platform-scripture 920; all nine no-unasked-edits e2e specs (including the four invisible-characters specs) passed on the first attempt (--retries=0, one run each).
    Earlier, at 870db8ab (the view registry, read-only sweep and paste fix merged in, CHANGELOG entry added, dist rebuilt) and paranext-core 68eefc1a31b, with core consuming the editor staged from 870db8ab:

  • packages/platform default run 4995 passed, 64 expected failures (pins), 126 skipped (245 s, 4 workers); with library-only views (SE_TEST_LIBRARY_VIEWS=1) 6407 passed, 71 expected failures, 24 skipped (442 s); libs/shared 920; libs/shared-react 2328 passed, 1 skipped; packages/utilities 89.

  • nx run-many -t lint typecheck -p shared shared-react platform-editor utilities 0 errors (one existing warning); verify-committed-dist.mjs ✓ after the dist commit, no stray Editorial.d.ts, index.d.ts unchanged.

  • paranext-core: typecheck and lint clean; core 7470 passed, 1 skipped; extensions 7422 (platform-scripture-editor 5528, platform-scripture 920); platform-bible-utils 736; platform-bible-react 2297 (unit 1659, storybook 638); dotnet 2962 passed, 14 skipped.

  • e2e no-unasked-edits (--retries=0, one worker, one run): power-standard, simple-paragraph-structure, power-preview, footnotes-pane-power-standard and footnotes-pane-simple all passed on the first attempt.

  • At 8163d705 (Standard-with-expanded-notes rows moved to the app's views): packages/platform default run 4984 passed, 64 expected failures, 142 skipped (257 s, 4 workers, idle machine); with SE_TEST_LIBRARY_VIEWS=1 6488 passed, 71 expected failures, 28 skipped (587 s, measured while an e2e run shared the machine); platform-editor lint and typecheck 0 errors (the existing warning); test-only, verify-committed-dist.mjs ✓ with dist unchanged.

Merging main: main's PT-4885 caret host already tags every host commit history-merge, so this branch's $tagCaretHostCommit gave way to it (the Undo/Redo tests still pass), and main's new mountViewEditor test helper now mounts through this branch's mountInView.

Earlier runs: ccdd06db (all gates green; Standard and Simple e2e passed, the old Power formatted spec failing on main's read-only Preview, since replaced in core), 5dd6f6df (all gates and e2e green on the first attempt), c48eda31 (e2e passed against core 836cc95073f), a7c8a714 after the first code review, d2c0a6f5 after the second (with main merged), a02d0046 (the paste and delta-op fixes; e2e passed first attempt in all three views), and 57e82b05 (the marker-apply and lone-~ follow-ups; the Standard e2e failed its first attempt and passed on rerun), each with the editor tests, lint, typecheck and the dist check green and core restaged and green.

Risk Level

Medium. The typing, NBSP and clipboard fixes change editor behavior, mostly in the hidden-marker views. Each is pinned by gesture-level tests and the oracles, and the riskiest fixes were independently reviewed. The attribute-value change touches the settle path (adaptor, display-run syncs, tokenizer), so it is the one to watch; the typing oracle and pending/settled checks pass. The composition fix can't be fully tested with a real input method in jsdom: its document result is tested with a simulated composition.

🤖 Generated with Claude Code


This change is <img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>


Generated by Claude Code

tjcouch-sil and others added 4 commits October 20, 2026 18:42
usjToUsxString(usj, { usxVersion }) stamps the requested version on the
root element instead of always writing USX_VERSION. USJ cannot carry the
source USX version (the editor regenerates usj.version everywhere), so a
host that needs its USX back at the version ParatextData served it must
say so explicitly. Default output is byte-identical to before.

Part of the PT-4197 data-fidelity audit; lets paranext-core drop its
post-hoc version="3.1" string replacement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pins the full load/save trip over the real per-chapter USX the paranext-core
PDP produces (vendored under testUsfmCorpus/pdp-usx/):

- corpus-pdp-usx-composed-round-trip: USX -> USJ -> mounted editor -> USJ ->
  USX per capture, in Standard, formatted and paragraph-structure views, with
  a pinned expected-divergence list (an unlisted divergence fails; a listed
  one that stops happening fails).
- testusfm-usx-usj-usx-round-trip: the USX-first converter net, so an
  asymmetric usx<->usj pair cannot hide from both directions.
- corpus-testusfm-round-trip and corpus-transform-fixed-point now run every
  fixture in all three editing views (they were Standard-only).
- New fixtures testUSFM-2SA-4 (NBSP shapes, verse bridges/segments, \fe, \x,
  \cl/\cd, callers, \ref), testUSFM-2SA-5 (damaged-but-at-rest: nameless
  markers, double \ca, chapter-line text) and testUSFM-1SA-1 (book intro),
  with .usj derived by the documented recipe (provider USX + usxStringToUsj;
  2SA-5 hand-corrected to the intended marker:"" shape).

Known losses are routed through each suite's expected-failure mechanism,
attributed to scripture-editors #11, #19 or PT-4391, so a merge that fixes
one flips the pin visibly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Only the committed utilities dist conflicted; rebuilt from the merged source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SE #11 fixed the leading-NBSP strip in a char span, the double-\ca fold in
the tokenizer, and the nameless-marker converter bug, so their pins flipped
as designed. The 2SA-5 .usj no longer needs a hand edit: the converter now
writes exactly what it holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tjcouch-sil and others added 25 commits October 22, 2026 17:20
…(PT-4391)

Outside Standard view a data NBSP is held as a literal NBSP, so a content
string that is exactly one NBSP (`\nd ~\nd*`, `\p ~`, a `~` between two
spans or after a note) was byte-identical to the engine's lone-NBSP display
nodes and the editor->USJ conversion dropped it, already at load.

The conversion now drops a lone NBSP only in the slots the engine's display
nodes occupy: right after an editable glyph, right after a read-only glyph
inside a span, first in a span or note, after a collapsed note's caller or
span, or inside an attribute run (tagged/token nodes and the mark cases are
unchanged). The one slot an author's `~` shares with a display node - a
span's first content in the marker modes without a separator, where the
empty-span placeholder sits - is disambiguated by a new node state,
`authoredNbspState`, that the forward adaptor's createChar sets.

The same rule is the live `$isLoneNbspContent` (node.utils.ts), used by the
logical content model and the Tier-2 fragment flattening so annotation
positions and settles agree with the export. CharNodePlugin no longer treats
an authored `~` as an empty span's placeholder when text is typed beside it.

Flips the pinned drops: constructed-usj-round-trip's paragraph-leading lone
NBSP, the 2SA-4 entries in the composed PDP round trip and the testUSFM
round trip (adaptor and transform legs), the formatted-view leading-NBSP pin,
and the logical-model spacer pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every per-view list the editor suites iterate is now a projection of
EDITOR_VIEWS in editorViews.test-helpers.tsx: the location oracle's
ORACLE_VIEWS/oracleView, the corpus suites' ROUND_TRIP_VIEW_CONFIGS and
EDITING_VIEW_CONFIGS (moved out of corpus-data.ts, which is compiled
with the library and so cannot import a test helper), and the new
EDITING_VIEWS (standard, formatted, paragraph-structure).

The two private Editor mounts (settledGetUsj's mountEditor and the
location oracle's mountInView) merge into one mountInView(usj, view,
opts); mountStandardViewEditor/mountExpandedNoteEditor are thin
wrappers. Also adds footnotePopoverView(host) (mirrors paranext-core's
FootnoteEditor options), forEachView, knownFailureRunner, and
noteEditorRef's optionsInView(view).

The Standard-with-expanded-notes round-trip label is now its registry
name, standard+expandedNotes (was standard-expanded).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PT-4197)

logicalModel.oracle gains formatted and paragraph-structure (no
separators); canonicalLocations gains paragraph-structure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-4197)

note-ops-popover-roundtrip mounts its host in Standard, formatted and
paragraph-structure, and its popover with footnotePopoverView(host), the
options paranext-core's FootnoteEditor builds (so the Standard popover
now also carries isNoteShellEditable: false, as the app's does). Glyph
shape and the Tier-2 fixed point are asserted only over an editable
host; ops/USJ idempotence and the Save write-back hold in all three.

New: Enter inside the popover's note content. It keeps the note whole
in every view, but over a hidden-marker host (formatted,
paragraph-structure) Enter is Lexical's own paragraph split: the caret
leaves the note for a new scaffolding paragraph, so the keystrokes that
follow are outside the note and the popover's Save (note op only) drops
them. Pinned as known failures.

knownFailureRunner now returns a registrar that appends the reason to
the test name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(PT-4197)

corpus-transform-fixed-point and the testUSFM transform leg mounted only
CharNodePlugin, MarkerEditPlugin and TextSpacingPlugin, so in formatted
and paragraph-structure the transforms those views actually run (the
note, paragraph and paragraph-marker-prefix plugins) never ran. Both now
mount the real Editor in each editing view through mountInView, dirty
every node once and read the document back with the adaptor Editor uses
on a local change. The composed provider-USX suite shares the same
runEveryTransform helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng view (PT-4197)

exactAnnotationLocations rows that stopped at standard/formatted (figure
caption hold, undo/redo, comment inside a caption, an NBSP inside a char
span, labels at a range's end) also run in paragraph-structure.
undoReachesFile's plain-text undo/redo control runs in all three
editing views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
navigationThenEdit and scrRefNavigationThenEdit run in Standard,
formatted and paragraph-structure: an edit right after a navigation
must still reach onUsjChange (a lost onUsjChange is a lost save).

Each row starts from no document selection. Without that, a row that
follows another inherits jsdom's stale selection and stops detecting
the leak it pins: with a cursor-change tag forced onto the navigation's
caret placement, only every other row failed until the reset was added.
scrRefNavigationThenEdit also asserts the reported USJ carries the note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
onUsjChangeSettled's view-independent rows run in Standard, formatted
and paragraph-structure: once per content commit (a new plain-text
row; the attribute row stays Standard's), nothing for a selection-only
commit or a commit that changes no bytes (after a load, a reload, and a
paragraph touch after a lossless or double-space load), a local
applyUpdate announced once, and every remote applyUpdate row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (PT-4197)

The verse-spanning-delete undo regressions (guarded and native) and the
removeCharacterMarker/replaceCharacterMarker/extendCharacterMarker ref
rows that mounted the library's default (formatted) view now run in
Standard, formatted and paragraph-structure.

Two extendCharacterMarker rows fail in Standard view and are pinned as
known failures: coalescing two \bd runs leaves two adjacent \bd spans,
and extending \bd with \nd as a conflicting marker removes \nd but
never applies \bd while reporting success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noteIndexAndHighlight, applyUpdateDomSelection and noteLooseTextApply
mount through optionsInView(view) in Standard, formatted and
paragraph-structure. The one row whose premise is Standard's display of
an unclosed note's caller (plain caller text, not an immutable caller
element) stays editable-marker only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (PT-4197)

charStackParagraphSplit and paraWholeDeletion pin the marker-edit
engine on hand-built Standard trees. Each gains real-Editor rows over
Standard, formatted and paragraph-structure judged on the saved USJ:
Enter inside \nd (and inside \wj \+nd) keeps both halves styled, and a
deletion of a paragraph's text, or across a paragraph boundary, keeps
the content the selection did not cover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PT-4197)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The popover suite's known-failure map is a plain string map, and the
deletion rows' Delete keypress checks the editor root for null itself
(the marker-edit requireDefined admits null).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…PT-4391)

Follow-up review fixes to the lone-NBSP content rule outside Standard view.

- CharNodePlugin: merging adjacent same-attribute glyph-less spans no
  longer carries an empty span's placeholder into the merged content.
  Landing after a nested span it read as content (the regression), and
  after a text-ending or `~` span Lexical merged it into the text before
  (`x~`, `~~`). The empty side now gives up its children and a caret on
  its placeholder moves to the seam. An author-marked `~` is never
  treated as a placeholder.
- Remove-style ($removeCharNodeKeepingContent) unwraps an author-marked
  lone `~` instead of deleting the span with it.
- NoteNodePlugin stripped a leading NBSP from any note char text on any
  transform (`\ft ~x` saved `\ft x`; typing after `\ft ~` lost it). It now
  uses CharNodePlugin's placeholder rule, which drops only a byte typed
  over a span that held nothing but the unmarked placeholder.
- $isCharContentEmpty counted any NBSP-only text as empty, so a
  close-and-reopen after a `~` span's `~` dropped the span with its `~`
  (unformatted, formatted, paragraph-structure). It now strips only the
  separator and asks $isLoneNbspContent about a lone NBSP.

Tests: gesture-level guards through the real Editor in every view
(lone-nbsp-gestures), CharNodePlugin merge/typing and NoteNodePlugin
units, $isCharContentEmpty units, and live/serialized agreement rows
inside spans and notes (read-only glyphs, collapsed-caller notes,
author-marked nodes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the committed platform dist conflicted; rebuilt from the merged source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A document now leaves the converters and the editor at the version it
arrived at, instead of being re-stamped 3.1 and forced back to 3.0 by the
host.

- `usxStringToUsj` keeps the root `<usx version>` (USJ_VERSION only when the
  USX declares none or an empty one); `usjToUsxString` writes `usj.version`
  (USX_VERSION only when the USJ has none).
- `Usj.version` is widened from `typeof USJ_VERSION` to `string` — a public
  type change, recorded in the utilities CHANGELOG and api report.
- The `usxVersion` option on `usjToUsxString` (Unreleased only) is removed;
  the round-trip audit suites now round-trip the version with no help, and
  the testUSFM USJ -> USX -> USJ net compares whole documents, version
  included.
- The editor remembers the version of `defaultUsj` / each `setUsj` and
  stamps it on every USJ it builds (load, local commit, apply, stale
  refresh, settled read via `$settledUsj`). The reverse adaptor takes the
  version to stamp; internal scratch USJ fed to the forward adaptor is
  unchanged. Loading a non-USJ_VERSION document no longer warns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Outside Standard view nothing turns Enter in a note into a \fp break, so
Lexical split the paragraph holding the note and left the caret outside
it; what was typed next was lost on Save. NoteEnterGuardPlugin claims
KEY_ENTER, INSERT_PARAGRAPH and INSERT_LINE_BREAK at critical priority
while the selection touches an expanded note, in every markerMode but
"editable".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as Paratext 9

Standard view saved a space typed directly after a char span's opener as
content (`\nd  things\nd*` -> content " things"). Paratext 9 skips all
whitespace after any opening marker (UsfmToken.Tokenize), so its Standard
view saves `\nd things\nd*`, and so does ParatextData on Platform.Bible's own
save path (ConvertUsxToUsfm runs NormalizeUsfm) — the editor's getUsj() was
the only place the space survived, so the editor reported a document the
file never held.

The span's content now drops its leading plain spaces on save, the rule a
paragraph's start and the text after a verse number already follow: the
screen keeps the typed space and the caret after it; the file does not get
it. The logical position model drops the same spaces so positions agree with
getUsj(). A span holding nothing but spaces keeps them (a selected space
wrapped in a span), since dropping that would join the words around it.

Catalog row O-5 (PT-4197). Pins updated deliberately: typedSeparatorSpace,
separatorGapSettle, verseAdjacentTyping, charAttributeTypedSettle (Standard
only; unformatted keeps whitespace), the adaptor caller-lookalike case, the
logical-model unit and oracle tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an while pending

When a settle creates a verse directly after a char span (e.g. a `\` typed in
front of `v 2 b` after `\w grace\w*`), TextSpacingPlugin's verse transform
puts a space between them once the verse lands. The read-only settle behind
a pending getUsj() creates no live nodes, so it reported the span and the
verse touching, then the same bytes came back spaced after the settle.

The read-only settle now mirrors the transform over the output of every
paragraph or chapter scope it re-tokenizes. A scope the live settle only
renames in place (signature unchanged) re-creates no verse, so it is left
alone on both sides.

Paratext 9 has this space too: UsfmToken.Tokenize inserts one between a
closing marker and a following \v on every read, and the file is the same
with or without it (the writer puts a line break before \v) — measured with
ParatextData 9.5.0.24. Catalog row O-7 (PT-4197).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the committed platform dist conflicted; rebuilt from the merged source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erse spacing

Only the committed platform dist conflicted; rebuilt from the merged source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lexical inserts a paste's nodes (internal flavor, or text/html, which is what
Ctrl+V delivers) with RangeSelection.insertNodes, which splits every inline
ancestor of the caret up to the paragraph: text pasted inside \nd Lord\nd* split
the span and landed between the halves, and text pasted inside a note landed in
the paragraph after it (cut + paste back inside a note moved the text out).

A SELECTION_INSERT_CLIPBOARD_NODES_COMMAND handler now inserts a payload of plain
prose as text when the paste starts in ordinary text inside an inline element,
the way typing it would. Structured payloads are still Lexical's.

Found by the PT-4197 clipboard corpus sweep (finding CB-2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In the views without char glyphs a span's lone NBSP is also the empty-span caret
placeholder, which the save drops unless the text is tagged as the author's. Only
the USJ-to-editor adaptor tagged it, so cutting or deleting `trail` from
`\nd trail~\nd*` saved an empty span: the NBSP outside the selection was lost.

A CharNodePlugin transform now tags a lone-NBSP text in a glyph-less span as
authored when that text held more than the placeholder in the last committed
state.

Found by the PT-4197 clipboard corpus sweep (finding CB-4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Standard view's USFM clipboard has a corpus sweep; formatted and
paragraph-structure had none. The new suite samples whole verses, ranges
starting or ending inside a char span, ranges holding a footnote, crossing a
paragraph boundary or a verse number, spanning a paragraph, and inside note
content, then copies and pastes (all flavors, html+plain as Ctrl+V delivers,
plain), cuts, cuts and pastes back, and pastes USFM-looking text, checking each
result and its undo/redo against a USJ-level model.

Pinned with knownFailureRunner (68 rows):
- paragraph-structure: a copy across a paragraph boundary carries the gutter
  glyph (\q1) in text/plain, so CommandMenuPlugin refuses its paste (CB-1);
- hidden views, html flavor: char attributes (closed) are not carried and a
  note's display spacers come back as text (CB-3);
- Standard reference rows over damaged USFM (2SA-5) and one derived-sid loss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tjcouch-sil and others added 2 commits November 2, 2026 13:38
… OT rules

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd separator ops reverts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@lyonsil lyonsil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A third review of this PR's source changes, at 314da06, with 5 findings. Each comment shows the severity it was settled at and a status: checked and confirmed means the finding was verified against the code at this head, and needs a human call means it could not be settled either way and is yours to judge.

(AI-assisted, with my guidance)

tjcouch-sil and others added 12 commits November 2, 2026 18:27
…its without a keydown

InlineRangeEditPlugin is mounted ahead of StructureKeyboardPlugin and claimed
text insertion and character delete first, removing every leaf in a range
including a verse marker. It now leaves such a range to the structure guards:
under protected for any range that blocks replacement, under guarded for an
unconfirmed delete of a range holding a verse marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h after it is deleted

The blank line took the removed paragraph's marker on any user edit. It now
does so only when content joined it or the removed paragraph held content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A paste spelled every no-break space not after an opening marker as `~`,
attribute lists included, where the tokenizer reads `~` in a value as a
literal. A no-break space between a char span's `|` and its closer, and one
pasted within a single attribute node, now takes the attribute spelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…changed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ribute paste and version doc changes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in protected mode keeps the verse

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…` again with the setting off

This reverts commit dd7abad.

With allowInvisibleCharacters off, ParatextData turns every U+00A0 in the
file into `~` when it reads it (ScrText.FixNBSP), attribute values
included, and Paratext 9 pastes every U+00A0 as `~`. Keeping a pasted
U+00A0 in a value only lasted until the next read, so a paste spells it
`~`, as Paratext 9 does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…evert

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… unclosed spans

The converter net now discovers every corpus oracle at run time, so 2SA-4, 2SA-5, 1SA-1 and
spacing-word-break are covered too (all pass).

A new suite types, backspaces and splits the paragraph inside a span the USFM never closed
(closed="false", to the paragraph end or closed implicitly by the next char marker), and types
before it, in every editing view, and checks that every span still saves as unclosed in both
the USJ and the USX the provider receives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	packages/platform/dist/index.js.map

@lyonsil lyonsil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A fourth review of this PR's source changes, at 6e2473a, with 3 findings. Each comment shows the severity it was settled at and a status: checked and confirmed means the finding was verified against the code at this head, and needs a human call means it could not be settled either way and is yours to judge.

(AI-assisted, with my guidance)

tjcouch-sil and others added 6 commits November 3, 2026 12:29
… command

Under structure protection "protected", an HTML paste or drop was sanitized and then inserted
with `selection.insertNodes` directly, so the listeners on
SELECTION_INSERT_CLIPBOARD_NODES_COMMAND never ran: pasted prose split the caret's char span
(`\nd Lo\nd*X\nd rd\nd*`), a paste over a range across a span edge landed outside the spans,
and pasted display annotations were not stripped. The sanitized nodes now go through the
command, falling back to a direct insert only when no listener claims it.

pasteOverInlineRange.test.tsx now runs every case under each protection mode, plus a paste at
a caret inside a span.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…one in a span

In the views that hide markers, a no-break space pasted or typed into existing text in a char
span was never marked as the author's, so when Enter split it off on its own, the lone NBSP read
as the empty-span placeholder and was dropped on save (`\nd Lord~\nd*` + Enter before the `~`
saved an empty `\nd \nd*`). A text holding an NBSP among other text is now marked too, and a
split copies the mark; an empty span's own placeholder is still left unmarked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to the copy limit

In the views that hide markers with `allowInvisibleCharacters` off, a copy that fits writes each
stored no-break space as `~`, but a copy or cut shortened to the copy limit wrote them as
U+00A0. The over-limit path now respells them the same way, counting only the text it writes
(a collapsed note's content is left out of both). The helpers move to
whitespaceDisplay.plugin.utils.ts so both plugins share them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ttle in them

usxStringToUsj means to drop the derived `vid` and `status` attributes, but it tested them for
truthiness, so an empty one survived. ParatextData writes `vid=""` on every paragraph that
continues a verse in a chapter after the first (a `\q2` line, a `\p` that starts mid-verse), and
Standard view's marker engine leaves a paragraph carrying an attribute it does not know as it
is: a `\nd ` or `\v ` typed into such a line stayed plain text until the chapter reloaded. Both
attributes are now dropped whatever their value.

The corpus oracles generated by this converter (2SA-4, 2SA-5, Matthew 5) lose their empty
`vid`s to match, and the USJ-first net no longer needs to tolerate it. Six Standard-view
clipboard rows on 2SA-5 that were pinned to PT-4388 now pass: the paragraphs they pasted into
carried `vid`. The pin keeps only the rows that copy 2SA-5's nameless paragraph marker.

continuationParagraphMarkers.test.tsx loads Matthew 2:6 as the provider serves it and checks
that `\nd ` typed key by key, or committed by the marker palette, settles in the `\q2` line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@lyonsil lyonsil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@lyonsil reviewed 237 files and all commit messages, made 1 comment, and resolved 24 discussions.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on tjcouch-sil).

lyonsil

This comment was marked as duplicate.

@tjcouch-sil
tjcouch-sil merged commit fbaf96c into main Nov 3, 2026
6 checks passed
@tjcouch-sil
tjcouch-sil deleted the pt-4197-data-fidelity branch November 3, 2026 20:40
lyonsil pushed a commit that referenced this pull request Nov 4, 2026
…rpus (#29)

* feat(utilities): let usjToUsxString stamp a caller-specified USX version

usjToUsxString(usj, { usxVersion }) stamps the requested version on the
root element instead of always writing USX_VERSION. USJ cannot carry the
source USX version (the editor regenerates usj.version everywhere), so a
host that needs its USX back at the version ParatextData served it must
say so explicitly. Default output is byte-identical to before.

Part of the PT-4197 data-fidelity audit; lets paranext-core drop its
post-hoc version="3.1" string replacement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: composed PDP round trip, USX-first net, corpus extension (PT-4197)

Pins the full load/save trip over the real per-chapter USX the paranext-core
PDP produces (vendored under testUsfmCorpus/pdp-usx/):

- corpus-pdp-usx-composed-round-trip: USX -> USJ -> mounted editor -> USJ ->
  USX per capture, in Standard, formatted and paragraph-structure views, with
  a pinned expected-divergence list (an unlisted divergence fails; a listed
  one that stops happening fails).
- testusfm-usx-usj-usx-round-trip: the USX-first converter net, so an
  asymmetric usx<->usj pair cannot hide from both directions.
- corpus-testusfm-round-trip and corpus-transform-fixed-point now run every
  fixture in all three editing views (they were Standard-only).
- New fixtures testUSFM-2SA-4 (NBSP shapes, verse bridges/segments, \fe, \x,
  \cl/\cd, callers, \ref), testUSFM-2SA-5 (damaged-but-at-rest: nameless
  markers, double \ca, chapter-line text) and testUSFM-1SA-1 (book intro),
  with .usj derived by the documented recipe (provider USX + usxStringToUsj;
  2SA-5 hand-corrected to the intended marker:"" shape).

Known losses are routed through each suite's expected-failure mechanism,
attributed to scripture-editors #11, #19 or PT-4391, so a merge that fixes
one flips the pin visibly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: drop the pins SE #11 fixed (PT-4197)

SE #11 fixed the leading-NBSP strip in a char span, the double-\ca fold in
the tokenizer, and the nameless-marker converter bug, so their pins flipped
as designed. The 2SA-5 .usj no longer needs a hand edit: the converter now
writes exactly what it holds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep an authored lone-NBSP content string outside Standard view (PT-4391)

Outside Standard view a data NBSP is held as a literal NBSP, so a content
string that is exactly one NBSP (`\nd ~\nd*`, `\p ~`, a `~` between two
spans or after a note) was byte-identical to the engine's lone-NBSP display
nodes and the editor->USJ conversion dropped it, already at load.

The conversion now drops a lone NBSP only in the slots the engine's display
nodes occupy: right after an editable glyph, right after a read-only glyph
inside a span, first in a span or note, after a collapsed note's caller or
span, or inside an attribute run (tagged/token nodes and the mark cases are
unchanged). The one slot an author's `~` shares with a display node - a
span's first content in the marker modes without a separator, where the
empty-span placeholder sits - is disambiguated by a new node state,
`authoredNbspState`, that the forward adaptor's createChar sets.

The same rule is the live `$isLoneNbspContent` (node.utils.ts), used by the
logical content model and the Tier-2 fragment flattening so annotation
positions and settles agree with the export. CharNodePlugin no longer treats
an authored `~` as an empty span's placeholder when text is typed beside it.

Flips the pinned drops: constructed-usj-round-trip's paragraph-leading lone
NBSP, the 2SA-4 entries in the composed PDP round trip and the testUSFM
round trip (adaptor and transform legs), the formatted-view leading-NBSP pin,
and the logical-model spacer pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: one shared registry of editor views and one mountInView (PT-4197)

Every per-view list the editor suites iterate is now a projection of
EDITOR_VIEWS in editorViews.test-helpers.tsx: the location oracle's
ORACLE_VIEWS/oracleView, the corpus suites' ROUND_TRIP_VIEW_CONFIGS and
EDITING_VIEW_CONFIGS (moved out of corpus-data.ts, which is compiled
with the library and so cannot import a test helper), and the new
EDITING_VIEWS (standard, formatted, paragraph-structure).

The two private Editor mounts (settledGetUsj's mountEditor and the
location oracle's mountInView) merge into one mountInView(usj, view,
opts); mountStandardViewEditor/mountExpandedNoteEditor are thin
wrappers. Also adds footnotePopoverView(host) (mirrors paranext-core's
FootnoteEditor options), forEachView, knownFailureRunner, and
noteEditorRef's optionsInView(view).

The Standard-with-expanded-notes round-trip label is now its registry
name, standard+expandedNotes (was standard-expanded).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the position oracles in formatted and paragraph-structure (PT-4197)

logicalModel.oracle gains formatted and paragraph-structure (no
separators); canonicalLocations gains paragraph-structure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the footnote popover round trip over every editing host (PT-4197)

note-ops-popover-roundtrip mounts its host in Standard, formatted and
paragraph-structure, and its popover with footnotePopoverView(host), the
options paranext-core's FootnoteEditor builds (so the Standard popover
now also carries isNoteShellEditable: false, as the app's does). Glyph
shape and the Tier-2 fixed point are asserted only over an editable
host; ops/USJ idempotence and the Save write-back hold in all three.

New: Enter inside the popover's note content. It keeps the note whole
in every view, but over a hidden-marker host (formatted,
paragraph-structure) Enter is Lexical's own paragraph split: the caret
leaves the note for a new scaffolding paragraph, so the keystrokes that
follow are outside the note and the popover's Save (note op only) drops
them. Pinned as known failures.

knownFailureRunner now returns a registrar that appends the reason to
the test name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the corpus transform legs through the real Editor per view (PT-4197)

corpus-transform-fixed-point and the testUSFM transform leg mounted only
CharNodePlugin, MarkerEditPlugin and TextSpacingPlugin, so in formatted
and paragraph-structure the transforms those views actually run (the
note, paragraph and paragraph-marker-prefix plugins) never ran. Both now
mount the real Editor in each editing view through mountInView, dirty
every node once and read the document back with the adaptor Editor uses
on a local change. The composed provider-USX suite shares the same
runEveryTransform helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: annotation and undo rows in paragraph-structure and every editing view (PT-4197)

exactAnnotationLocations rows that stopped at standard/formatted (figure
caption hold, undo/redo, comment inside a caption, an NBSP inside a char
span, labels at a range's end) also run in paragraph-structure.
undoReachesFile's plain-text undo/redo control runs in all three
editing views.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: navigation-then-edit save pins in every editing view (PT-4197)

navigationThenEdit and scrRefNavigationThenEdit run in Standard,
formatted and paragraph-structure: an edit right after a navigation
must still reach onUsjChange (a lost onUsjChange is a lost save).

Each row starts from no document selection. Without that, a row that
follows another inherits jsdom's stale selection and stops detecting
the leak it pins: with a cursor-change tag forced onto the navigation's
caret placement, only every other row failed until the reset was added.
scrRefNavigationThenEdit also asserts the reported USJ carries the note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: host change-notification rows in every editing view (PT-4197)

onUsjChangeSettled's view-independent rows run in Standard, formatted
and paragraph-structure: once per content commit (a new plain-text
row; the attribute row stays Standard's), nothing for a selection-only
commit or a commit that changes no bytes (after a load, a reload, and a
paragraph touch after a lossless or double-space load), a local
applyUpdate announced once, and every remote applyUpdate row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: Editor undo and character-marker ref rows in every editing view (PT-4197)

The verse-spanning-delete undo regressions (guarded and native) and the
removeCharacterMarker/replaceCharacterMarker/extendCharacterMarker ref
rows that mounted the library's default (formatted) view now run in
Standard, formatted and paragraph-structure.

Two extendCharacterMarker rows fail in Standard view and are pinned as
known failures: coalescing two \bd runs leaves two adjacent \bd spans,
and extending \bd with \nd as a conflicting marker removes \nd but
never applies \bd while reporting success.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: note EditorRef suites in every editing view (PT-4197)

noteIndexAndHighlight, applyUpdateDomSelection and noteLooseTextApply
mount through optionsInView(view) in Standard, formatted and
paragraph-structure. The one row whose premise is Standard's display of
an unclosed note's caller (plain caller text, not an immutable caller
element) stays editable-marker only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: paragraph split and deletion content rows in every editing view (PT-4197)

charStackParagraphSplit and paraWholeDeletion pin the marker-edit
engine on hand-built Standard trees. Each gains real-Editor rows over
Standard, formatted and paragraph-structure judged on the saved USJ:
Enter inside \nd (and inside \wj \+nd) keeps both halves styled, and a
deletion of a paragraph's text, or across a paragraph boundary, keeps
the content the selection did not cover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: word the navigation row-isolation comment as the hazard it is (PT-4197)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: satisfy the typecheck in two per-view rows (PT-4197)

The popover suite's known-failure map is a plain string map, and the
deletion rows' Delete keypress checks the editor root for null itself
(the marker-edit requireDefined admits null).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: never save an empty span's placeholder or drop an authored `~` (PT-4391)

Follow-up review fixes to the lone-NBSP content rule outside Standard view.

- CharNodePlugin: merging adjacent same-attribute glyph-less spans no
  longer carries an empty span's placeholder into the merged content.
  Landing after a nested span it read as content (the regression), and
  after a text-ending or `~` span Lexical merged it into the text before
  (`x~`, `~~`). The empty side now gives up its children and a caret on
  its placeholder moves to the seam. An author-marked `~` is never
  treated as a placeholder.
- Remove-style ($removeCharNodeKeepingContent) unwraps an author-marked
  lone `~` instead of deleting the span with it.
- NoteNodePlugin stripped a leading NBSP from any note char text on any
  transform (`\ft ~x` saved `\ft x`; typing after `\ft ~` lost it). It now
  uses CharNodePlugin's placeholder rule, which drops only a byte typed
  over a span that held nothing but the unmarked placeholder.
- $isCharContentEmpty counted any NBSP-only text as empty, so a
  close-and-reopen after a `~` span's `~` dropped the span with its `~`
  (unformatted, formatted, paragraph-structure). It now strips only the
  separator and asks $isLoneNbspContent about a lone NBSP.

Tests: gesture-level guards through the real Editor in every view
(lone-nbsp-gestures), CharNodePlugin merge/typing and NoteNodePlugin
units, $isCharContentEmpty units, and live/serialized agreement rows
inside spans and notes (read-only glyphs, collapsed-caller notes,
author-marked nodes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: preserve the USX/USJ version end to end (PT-4197)

A document now leaves the converters and the editor at the version it
arrived at, instead of being re-stamped 3.1 and forced back to 3.0 by the
host.

- `usxStringToUsj` keeps the root `<usx version>` (USJ_VERSION only when the
  USX declares none or an empty one); `usjToUsxString` writes `usj.version`
  (USX_VERSION only when the USJ has none).
- `Usj.version` is widened from `typeof USJ_VERSION` to `string` — a public
  type change, recorded in the utilities CHANGELOG and api report.
- The `usxVersion` option on `usjToUsxString` (Unreleased only) is removed;
  the round-trip audit suites now round-trip the version with no help, and
  the testUSFM USJ -> USX -> USJ net compares whole documents, version
  included.
- The editor remembers the version of `defaultUsj` / each `setUsj` and
  stamps it on every USJ it builds (load, local commit, apply, stale
  refresh, settled read via `$settledUsj`). The reverse adaptor takes the
  version to stamp; internal scratch USJ fed to the forward adaptor is
  unchanged. Loading a non-USJ_VERSION document no longer warns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Swallow Enter inside an expanded note where markers are not editable

Outside Standard view nothing turns Enter in a note into a \fp break, so
Lexical split the paragraph holding the note and left the caret outside
it; what was typed next was lost on Save. NoteEnterGuardPlugin claims
KEY_ENTER, INSERT_PARAGRAPH and INSERT_LINE_BREAK at critical priority
while the selection touches an expanded note, in every markerMode but
"editable".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(platform-editor): save no space typed right after a char opener, as Paratext 9

Standard view saved a space typed directly after a char span's opener as
content (`\nd  things\nd*` -> content " things"). Paratext 9 skips all
whitespace after any opening marker (UsfmToken.Tokenize), so its Standard
view saves `\nd things\nd*`, and so does ParatextData on Platform.Bible's own
save path (ConvertUsxToUsfm runs NormalizeUsfm) — the editor's getUsj() was
the only place the space survived, so the editor reported a document the
file never held.

The span's content now drops its leading plain spaces on save, the rule a
paragraph's start and the text after a verse number already follow: the
screen keeps the typed space and the caret after it; the file does not get
it. The logical position model drops the same spaces so positions agree with
getUsj(). A span holding nothing but spaces keeps them (a selected space
wrapped in a span), since dropping that would join the words around it.

Catalog row O-5 (PT-4197). Pins updated deliberately: typedSeparatorSpace,
separatorGapSettle, verseAdjacentTyping, charAttributeTypedSettle (Standard
only; unformatted keeps whitespace), the adaptor caller-lookalike case, the
logical-model unit and oracle tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(platform-editor): report the space before a verse after a char span while pending

When a settle creates a verse directly after a char span (e.g. a `\` typed in
front of `v 2 b` after `\w grace\w*`), TextSpacingPlugin's verse transform
puts a space between them once the verse lands. The read-only settle behind
a pending getUsj() creates no live nodes, so it reported the span and the
verse touching, then the same bytes came back spaced after the settle.

The read-only settle now mirrors the transform over the output of every
paragraph or chapter scope it re-tokenizes. A scope the live settle only
renames in place (signature unchanged) re-creates no verse, so it is left
alone on both sides.

Paratext 9 has this space too: UsfmToken.Tokenize inserts one between a
closing marker and a following \v on every read, and the file is the same
with or without it (the writer puts a line break before \v) — measured with
ParatextData 9.5.0.24. Catalog row O-7 (PT-4197).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(platform-editor): rebuild committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep pasted prose inside the char span or note it is pasted into

Lexical inserts a paste's nodes (internal flavor, or text/html, which is what
Ctrl+V delivers) with RangeSelection.insertNodes, which splits every inline
ancestor of the caret up to the paragraph: text pasted inside \nd Lord\nd* split
the span and landed between the halves, and text pasted inside a note landed in
the paragraph after it (cut + paste back inside a note moved the text out).

A SELECTION_INSERT_CLIPBOARD_NODES_COMMAND handler now inserts a payload of plain
prose as text when the paste starts in ordinary text inside an inline element,
the way typing it would. Structured payloads are still Lexical's.

Found by the PT-4197 clipboard corpus sweep (finding CB-2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the no-break space a cut leaves as a char span's only text

In the views without char glyphs a span's lone NBSP is also the empty-span caret
placeholder, which the save drops unless the text is tagged as the author's. Only
the USJ-to-editor adaptor tagged it, so cutting or deleting `trail` from
`\nd trail~\nd*` saved an empty span: the NBSP outside the selection was lost.

A CharNodePlugin transform now tags a lone-NBSP text in a glyph-less span as
authored when that text held more than the placeholder in the last committed
state.

Found by the PT-4197 clipboard corpus sweep (finding CB-4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Sweep copy, cut and paste over the testUSFM corpus in every editing view

Standard view's USFM clipboard has a corpus sweep; formatted and
paragraph-structure had none. The new suite samples whole verses, ranges
starting or ending inside a char span, ranges holding a footnote, crossing a
paragraph boundary or a verse number, spanning a paragraph, and inside note
content, then copies and pastes (all flavors, html+plain as Ctrl+V delivers,
plain), cuts, cuts and pastes back, and pastes USFM-looking text, checking each
result and its undo/redo against a USJ-level model.

Pinned with knownFailureRunner (68 rows):
- paragraph-structure: a copy across a paragraph boundary carries the gutter
  glyph (\q1) in text/plain, so CommandMenuPlugin refuses its paste (CB-1);
- hidden views, html flavor: char attributes (closed) are not carried and a
  note's display spacers come back as text (CB-3);
- Standard reference rows over damaged USFM (2SA-5) and one derived-sid loss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Rebuild the committed dist for the clipboard fixes

Carries the inline-prose paste and the remaining-NBSP transform into
packages/platform/dist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: typing-fidelity oracle harness for the hidden-marker views (PT-4197)

A keystroke oracle for formatted and paragraph-structure, where no
pending marker edit exists to settle. The expectation for each keystroke
is computed from the loaded USJ alone: each leaf block read as units (a
character, or an atom shown whole: verse, collapsed note, milestone,
figure), a caret as a unit index, and typing, Backspace, Delete, Enter
and typing over a range edited into the USJ tree there. The saved USJ
must be one of the expected documents, Undo must restore the loaded USJ
exactly and Redo re-apply the keystroke's.

Keystrokes arrive as a browser delivers them: the caret placed with a
selection change, the DOM selection made the editor's, a real keydown
on the root, then beforeinput and the text insertion. Selection.modify
(which jsdom lacks and Lexical's deleteCharacter needs) is supplied at
character granularity. Failures are reported and listed by shape.

Exports the pending-equals-settled oracle's keystroke corpus for reuse.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: Delete before a collapsed note no longer throws (PT-4197)

In the hidden-marker views, Delete just before a collapsed note removes
its caller; NoteNodePlugin then removed the note and went on to move its
first child out of the removed node, which threw ('Expected node to have
a parent'), discarding the edit. Return once the note is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: joining a paragraph into an empty \b keeps its text (PT-4197)

ParaNodePlugin refuses text typed into an empty blank line by clearing
it. A join with the paragraph after it (Backspace at that paragraph's
start, Delete in the blank line, or a range across the break typed over)
also lands content in the empty \b, and the clear deleted that whole
paragraph's text in the formatted view. Clear only when the paragraph
after the blank line is still there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: Undo takes back Enter in front of a char span or a note (PT-4197)

Lexical's history folds an update whose only dirty leaf is a text that
kept its text into the previous undo step, even when tagged as a push.
A paragraph break in front of an inline element is such an update in the
formatted view, so Undo did not take the break back. ParaNodePlugin now
marks a created paragraph's first leaf dirty as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: an author's no-break space left alone in a span is saved (PT-4197)

In the views without a span separator (hidden and visible markers), a
lone NBSP in a span is the empty-span placeholder unless it carries
authoredNbspState, which the adaptor set only on a span text that was
exactly one NBSP at load. Deleting the rest of \nd ~x\nd* (or x~)
left an unmarked NBSP, and the span saved empty; an NBSP typed into an
empty span lost to the placeholder drop the same way.

The adaptor now marks every span text holding an NBSP (Lexical keeps
node state through text edits and copies it on splitText), and the
placeholder drop marks the NBSP it leaves behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: typing over a range across a span edge or a note keeps the rest (PT-4197)

Lexical's RangeSelection.insertText moves the content after a range's
last text to after its first text, into the inline element the range
starts in, and resolves a range start at a text's end into the inline
element after it. In the hidden-marker views that pulled the text after
an enclosing char span inside the span (out of order), and, for a range
starting just before a collapsed note, put the typed text and the rest
of the range into the note, which was then removed with them.

InlineRangeEditPlugin (mounted for markerMode hidden) replaces a range
over more than one text of one paragraph itself: it cuts the edge texts,
removes the nodes between and the spans and notes that leaves empty,
and puts a point Lexical resolved into a collapsed note back beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: run the typing-fidelity oracle in formatted and paragraph-structure (PT-4197)

Two files per view (the keystroke corpus with the short testUSFM
chapters; the long testUSFM and WEB chapters), paragraph-structure under
Simple mode's guarded structure protection. The shapes still failing are
listed with their reasons: Redo lost or a second Undo needed around an
empty verse, one Enter-at-paragraph-start Undo in WEB Matthew 5, and one
ignored Backspace at an element caret in 1 Samuel 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build: rebuild the committed platform output (PT-4197)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(platform-editor): rebuild committed dist for the typing and clipboard merges

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* test(platform-editor): hold each keystroke to keeping content, and pin separator removal

The pending/settled oracle compared the pending document with the settled
one, so content a transform dropped in the keystroke's own update was
invisible: the pending document already lacked it and settled to itself.
A mutation that removes a span's following text once its separator is
deleted passed every check. Each edit's screen is now read as the edit
left it (before transforms) and as it landed, and the landed screen may
only respell the spliced one: no content byte (whitespace and marker
tokens aside) may go. Range keystrokes are excluded: the oracle replaces
a range with a direct insertText that skips the commands the marker-edit
engine answers, so their transforms are not what a keystroke does. All
four oracle views stay green with nothing new listed.

Catalog row O-8 (PT-4197) reported Backspace after a `\w` opener's
separator losing the span and the text after it. It does not: the bytes
on screen (`\p In the \wgrace|grace\w* of God`) settle, pending and
settled alike, to `\p In the ` and a new unknown paragraph `\wgrace`
holding the rest — ParatextData's own reading (an unknown marker is a
paragraph outside a note, a char span inside one; checked against its
USX). The report came from a probe that printed only the first block.
New cases pin it for Backspace and Delete across attributes, `\nd`,
nesting and a note, comparing the whole document with ParatextData's USX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(platform-editor): save a char span holding only spaces as empty, as Paratext 9

Standard view already dropped the spaces after a char opener's separator,
but kept them when they were all the span held (a selected space wrapped
in `\nd`). ParatextData skips all whitespace after an opener there too:
`one\nd  \nd*two` saves as `one\nd \nd*two` (USX `<char style="nd" />`),
so the words join in the file. The editor's saved USJ now matches; the
screen still shows the space inside the span. The logical position model
drops the exception with it, so positions agree with getUsj().

A no-break space is data and stays: `\nd ~lead\nd*` and `\nd ~\nd*` keep
it on load and with a space typed in front of it (new tests; ParatextData
keeps both too).

Pin updated deliberately: markerMenuApply's whitespace-only wrap now saves
`["one", {nd}, "two"]` while the live span still holds the space.

Catalog row O-5 follow-up (PT-4197).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform-editor): rebuild committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(shared-react): keep alike char spans apart once a deletion leaves them side by side

In the hidden-marker views (formatted, paragraph-structure) CharNodePlugin
combined two adjacent same-marker, same-attribute spans whenever either was
dirtied, so deleting the text between `\w one|lemma="x"\w*` and
`\w two|lemma="x"\w*`, or joining the paragraphs they end and start, saved one
`\w onetwo\w*` and changed word segmentation (typing oracle finding T-7).
Standard view keeps them apart (its spans carry glyphs), and so does
Paratext 9.

Two spans that were already alike, separate spans in the last committed state
now stay separate. Spans an update creates (a marker applied to a run, a
split's continuation, remote content) still combine with an alike neighbor,
as does a span whose marker an update changes to match.

- Regression tests in every editing view plus paragraph-structure (guarded):
  deleting the space between, and joining the paragraphs.
- CharNodePlugin unit tests that expect a combine now add their spans by an
  update once the plugin runs (they previously relied on the plugin's
  registration re-dirtying the initial state, i.e. spans the document already
  held); new unit test for the keep-apart case.
- Typing oracle: drop the "adjacent alike spans are one" allowance; both views'
  corpus and chapters pass without it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform-editor): rebuild committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* test(clipboard): name the tickets that fix the clipboard corpus pins

The paragraph-structure gutter-glyph copy pins (CB-1, 44 rows) name
PT-4911, and the hidden-view html-flavor pins (CB-3, 13 rows) name
PT-4921, in a TODO beside each rule and at the head of its reason. The
Standard-view damaged-USFM and derived-sid rows have no ticket yet and
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(shared-react): arm a chapter for the guarded two-press delete

Under "guarded" structure protection (Simple mode) a chapter marker beside the
caret was outside the two-press guard: Delete at the end of the block before
`\c N` removed the chapter on the first press with no arming, and Backspace at
the start of the paragraph after it armed that paragraph, whose merge then
did nothing because a paragraph cannot merge into a chapter (typing oracle
finding T-10).

$structuralDeleteTarget now resolves the atomic chapter glyph
(ImmutableChapterNode) beside a block edge as a "chapter" target. The first
press node-selects it (as for a verse) and the armed chapter carries the
verse-selected class, so it shows the armed styling; the second press removes
it and returns the caret where the first press found it. The opposite key
cancels. No host hint kind is published for a chapter (the host's hint copy is
verse/selection-only), matching paragraph merges. Standard view (editable
chapter text), "protected" (still refused) and "off" (Power mode) are
unchanged.

Tests: guardedChapterDeletion.test.tsx in formatted and paragraph-structure
views - Backspace after / Delete before `\c 1` and `\c 2` arm then delete
(guarded), cancel on the opposite key, refused under protected, native under
off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(platform-editor): keep a nameless character marker's empty name on export

ParatextData reads a bare `\` inside a note (`\f + \ft \ b\f*`) as
`<char style="" status="unknown">`. The converters already keep the empty
style as `marker: ""`, but the editor's export turned an empty char marker
into no marker at all, so getUsj() handed back a char with no name and the
USX written from it was not the USX loaded. createCharMarker now keeps
`marker: ""`, the way nameless paragraphs already round-trip.

A char with no marker at all in its USJ (not a shape USX can express) now
exports as `marker: ""`, as an absent paragraph marker already exports as `p`.

Tests: namelessChar.test.tsx loads ParatextData's captured USX for an unclosed
and a `\*`-closed nameless span in a footnote, and asserts getUsj() and the
USX written from it are unchanged in Standard, formatted and
paragraph-structure views; converter tests for a char's empty style.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* docs: correct what ParatextData does with a marker-bearing \cp

Comments claimed ParatextData keeps every `\cp` that holds a marker as its own
marker. Measured, it folds the text before the first marker into the
chapter's pubnumber and leaves the rest as loose content after the chapter;
only a `\cp` whose content starts with a marker stays a `\cp`. The editor
deliberately settles such a `\cp` to the USFM-spec shape: the saved bytes are
the ones Paratext 9 writes, and a reload shows Paratext 9's shape. No
behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform-editor): rebuild committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* feat(shared-react): publish a chapter hint kind for the guarded delete

An armed chapter in Simple mode's two-press delete now sets
data-verse-delete-kind="chapter" (with data-verse-delete-intent) on the
editor root, as an armed verse or selection does, so the host can show a
chapter-specific "press again" hint. A paragraph merge still publishes no
hint kind. CHANGELOG records the chapter arming and the new value.

Tests: guardedChapterDeletion (formatted + paragraph-structure) and the
StructureKeyboardPlugin DOM-signal suite expect kind=chapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* test(platform): Standard view may drop an empty derived sid on a verse paste

Pasting `\v 9 x` as plain text in Standard view rebuilds part of the
paragraph it lands in, and the 2SA-3 fixture's empty derived `sid`s on the
re-derived verses do not survive. Dropping derived metadata is accepted
(data-fidelity catalog F-8), so this is now an expectation rather than a
known failure: in the paragraph the verse lands in, an empty `sid` may be
dropped, and everything else (non-empty `sid`s included) must still match
exactly. The row still performs the paste, undo and redo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* test: name the tickets for the remaining untracked expected failures

- PT-4924: typing-oracle T-11 / T-12 / T-16 entries in both views'
  expected-failures JSON (in each reason).
- PT-4925: Standard view's extendCharacterMarker coalesce/conflict rows
  (EXTEND_KNOWN_FAILURES).
- PT-4388: the Standard-view 2SA-5 damaged-USFM clipboard rows and the
  fragment tokenizer's "testUSFM 2SA 5" known divergence (bare `\` as a
  nameless marker, waiting on ParatextData 9.6).

Every pin and what it asserts is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix: a join into an empty \b keeps the joined paragraph, as Paratext 9 (PT-4197)

Backspace at the start of a paragraph after an empty \b, Delete in the
blank line, or a range across the break typed over joined the paragraph
into the \b and kept the blank-line marker on text. Paratext 9's
formatted view joins paragraphs the way Gecko joins two blocks: the
second block survives (JoinNodeTransaction keeps the right node), so the
blank line goes and the paragraph keeps its own marker. ParaNodePlugin
now gives a blank \b that takes in a joined paragraph that paragraph's
marker and attributes, rewriting a gutter glyph to match, in formatted,
paragraph-structure and Simple's guarded two-press join. Standard view
edits marker text and is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* test: a typed U+00A0 saves as a space in Standard view, as Paratext 9 (PT-4197)

Paratext 9's Standard view (AllowInvisibleChars off) saves every U+00A0
in the editor as a space (UsfmHtmlUtilities.LegacyPostprocessUsfm) and
the author's no-break space is typed as `~`. Ours does the same; pin it
between letters, after a space and at a span's start, and pin `~` typed
saving an NBSP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build: rebuild committed dist for the \b join fix (PT-4197)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(shared-react): paragraph-structure Enter drops the break's space

Enter at a word break (`In| the`) left ` the` in the new paragraph in the
paragraph-structure (Simple) view, while Standard and formatted save
`the`. ParaNodePlugin trims a new paragraph's leading space from its first
child, which in that view is the gutter marker glyph rather than the text,
so nothing was trimmed. It now looks past a leading gutter glyph.

Tests: typingFidelity.regressions "Enter at a word break" in all three
editing views, unguarded and guarded (red in paragraph-structure: saved
" the beginning"). The typing oracle no longer accepts a new paragraph
that starts with text keeping its leading space (it fails 23 corpus rows
without the fix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform): rebuild the committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* feat(platform): ViewOptions.allowInvisibleCharacters keeps ~ and U+00A0 apart in Standard view

A Paratext project with AllowInvisibleChars on stores a no-break space as
U+00A0 itself and `~` as an ordinary character (measured with ParatextData:
load, save and round trip all carry both verbatim and distinct). Standard
view always applied the `~` = NBSP rules, so a literal `~` in such a
project was read as a no-break space and saved as U+00A0, and a U+00A0
typed there saved as a space.

With the new option on, a stored U+00A0 displays as NBSP_SYMBOL (U+237D)
instead of `~`, `~` stays an ordinary character, and every layer that
maps display or fragment text to data takes the spelling from
getNbspSpelling(viewOptions): the load/save adaptors, the tier-2 and
virtual settle tokenizer (new UsfmFragmentOptions.nbspSpelling), the
Unformatted content fragments, paste, copy (U+00A0 written as itself),
the Markers view's copy, and a typed U+00A0 (command and native
beforeinput routes). getViewMode ignores the field. Off is unchanged.

Tests: invisibleCharacters.test.tsx (load/save, edit, span and note,
typed ~, typed U+00A0 by both routes, paste, copy, settle and unsettled
read, Unformatted settle, Markers copy, formatted/paragraph-structure
pins), tokenizer nbspSpelling, getViewMode/getNbspSpelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform): rebuild the committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* feat(platform): toUsfmNbspSpelling for hosts that read the editor's display text

A host that copies or searches text read from the editor's DOM (the
footnote popover's Copy button, a Find seeded from the selection) sees
Standard view's display spelling of a no-break space. With
allowInvisibleCharacters on that is U+237D, which is not the project's
text; this turns it back into U+00A0, and leaves the text unchanged when
the option is off (`~` is the project's spelling then).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform): rebuild the committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* test(platform-editor): keep a note's empty caller empty through edits in every view

ParatextData reads `\f \ft x\f*` (one or two spaces after the marker) as
caller="" and writes it back as `\f \ft x\f*` (measured with the corpus
capture harness), and the tokenizer has followed that since PR #11. The
existing rows covered only the expanded editable views; this pins a
load, an edit beside the note and one inside it, and a departure in
every editor view, saving caller "".

Fails in all 8 views if the exporter writes "+" for an empty caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* feat(platform): U+001A is the no-break-space stand-in, drawn as a blank, and never leaves the editor

With ViewOptions.allowInvisibleCharacters on, Standard view held a stored
U+00A0 as U+237D (shown as a visible box glyph). U+237D can occur in
Paratext 9 text, so a literal one was saved as a no-break space. It now
holds U+001A (SUBSTITUTE), which ParatextData's ScrText.GetText turns
into a space on every read, so no project text can hold one.

- Rendering: a one-glyph font (empty, 0.25em wide, unicode-range U+1A,
  NBSP_STAND_IN_FONT_FAMILY) draws it as a blank the width of a space.
  The editor adds the face to its document through the CSS Font Loading
  API (from bytes, so no CSP font-src is involved). Chromium draws a
  control character as a box unless the FIRST family has a glyph for
  it, so generateUsjCss and usj-nodes.css lead every Scripture
  font-family with it.
- It never leaves the editor: delta ops given out (onUsjChange,
  getNoteOps, getOpsAfterNote, takeOpsAfterNote) carry U+00A0 for it and
  a space for an NBSP the view shows in a space run; applyUpdate and
  replaceEmbedUpdate take a U+00A0 that is not a marker separator as a
  stored no-break space; the internal Lexical clipboard flavor is left
  out when the selection holds a stand-in; a drag out carries the copy's
  USFM text and html; annotation callbacks and note-caller previews give
  U+00A0.
- Incoming text: every C0 control character but tab, LF and CR becomes a
  space in Standard view before anything reads it (load incl. attribute
  values, paste, drop, typing by both routes, applied ops).
- A drop into Standard view now goes in as a paste of the same data.
- The fragment tokenizer keeps the spelling as a character
  (regularizeSpaces turned control characters into spaces first).

Leaking before (measured against the U+237D build): ops out, note ops,
the footnote-editor round trip, ops in (U+00A0 saved as a space), the
Lexical clipboard flavor, the cut, the async paste, drag and drop, and
annotation text. Forced-on run (every registry view with the option):
only the 5 intended off-mode pins fail.

Known gap (both settings, not changed here): an attribute value's
U+00A0 is shown raw in its attribute run, and a settle of its paragraph
reads it as a space.

Tests: nbspStandIn.test.tsx (35), whitespaceDisplay.utils.test.ts
(control characters, ops mapping), usj-nodes.css and generateUsjCss
font stack, invisibleCharacters.test.tsx on the new stand-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(platform): a note caller's click hands out the note's ops with U+00A0

The getNoteOps a noteCallerOnClick handler receives read the note's ops
straight from the tree, so with allowInvisibleCharacters on they carried
the internal stand-in; Platform.Bible's footnote popover applied them,
and the incoming rule turned each stand-in into a space (seen in the
running app). They now go out through opsForHost like every other op.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform): rebuild the committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* Address review-paratext findings on the data-fidelity branch

- Fix InlineRangeEditPlugin: typing over a range whose two edge texts are
  both wholly selected no longer falls through to Lexical, which pulled the
  text after a char span into the span. The plugin now steps aside only for
  a whole-edge deletion. Regression rows added to
  typingFidelity.regressions.test.tsx.
- Add standardViewDrop.test.tsx: drop-as-paste rows under both
  allowInvisibleCharacters settings.
- Add the missing CHANGELOG entries (platform Fixed section; utilities note
  that the Scribe editor does not yet carry the USJ version through).
- Replace in-code Jira/branch references with forward-facing wording per
  CLAUDE.md, keeping open TODO(PT-x) markers.
- Document that allowInvisibleCharacters must be right before the first
  load; correct the vertical-tab/form-feed wording; clipboard-semantics
  doc updates.
- Cleanups: $collapsedNoteOf uses $findMatchingParent, ParaNodePlugin key
  lookup, CharNodePlugin TSDoc, un-export $isSelectionInExpandedNote,
  NoteEnterGuardPlugin ordering comment, corpus fixed-point test uses
  knownFailureRunner, reshaped half-flat test trees, comment re-wraps,
  demo armed-chapter CSS.
- Rebuild committed packages/platform/dist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session-URL: https://claude.ai/code/session_01KiNAWjh4KAwjzn1g34EePL

* fix(platform): an attribute value's no-break space keeps its spelling in Standard view

Standard view showed a U+00A0 inside an attribute value as a raw U+00A0
in its attribute run, which the engine reads as display spacing, so a
settle of the paragraph (Tier-2 rebuild or virtual settle) saved it as a
plain space, in both ViewOptions.allowInvisibleCharacters settings. The
value now takes the body text's spelling (`~`, or the U+001A stand-in
with the setting on) everywhere attribute state becomes display text:

- the adaptor's runs for char spans, milestones, \va/\vp, \cat, \ca/\cp
  and an UnknownNode's display parts (figure src, sidebar \cat, periph);
- the display-run syncs, which redraw from node state without view
  options: MarkerEditPlugin registers the editor's spelling
  (registerNbspDisplaySpelling / $getNbspDisplaySpelling in shared);
- the `|...` bytes an unwrapped or emptied span leaves as content, a
  collapsed note's copied \cat, and the annotation byte maps.

The tokenizer reads the spelling back as U+00A0 inside milestone
(including an absorbed list), figure and \periph attribute values, as it
already did for body text, char-span attributes and the attribute
markers. Unformatted view, which shows a value's U+00A0 as itself, now
respells it past the run's leading separator in its settle fragment
(attributeFragmentText), as it does content, instead of flattening it to
a space.

Tests: attributeValueNbsp.test.tsx (each producer shape in Standard view
with the option off and on: display, paragraph settle, value edit read
before and after the settle, copy and paste with every flavor and with
plain text alone; figure; span unwrap and paragraph-break emptying; a
collapsed note's copied \cat; Unformatted; formatted and
paragraph-structure pins) and tokenizer cases in
usfmFragmentToUsj.test.ts. The nbspStandIn chapter row now settles the
chapter line instead of pinning the raw display.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* build(platform): rebuild the committed dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDpA6DiFYGheLpPSaj6yPz

* fix(shared-react): rejoin a char span that Enter split when Backspace joins the paragraphs back

Keeping alike char spans apart once both are in the document also kept apart
the two halves of one span that Enter cut in two, so Enter then Backspace
inside `\nd thing\nd*` saved `\nd th\nd*\nd ing\nd*` in the views that hide
markers.

- CharNode.insertNewAfter records, per editor, which span each split
  continuation came from; $wereSplitFromOneSpan lets those pieces merge again.
  Spans the document held apart still stay apart.
- Regression test in every hidden-marker view: Enter inside a span, then
  Backspace, saves one span.
- InlineRangeEditPlugin: comment the composition gap (an input method started
  over a cross-span range takes Lexical's own path).
- CHANGELOG entry extended.
- Rebuild committed packages/platform/dist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session-URL: https://claude.ai/code/session_018WCZ5KDNh1Vqzt3JP6jCnp

* fix(shared-react): keep Undo/Redo working around empty paragraphs and caret hosts

- Undo after a guarded two-step removal of an empty paragraph (an empty
  `\b`) now brings it back. Lexical's history folds an update whose only
  dirty leaf is a text that kept its text into the step before, even one
  tagged history-push, so `$mergeParaIntoPrevious` marks the removed
  paragraph's leaves dirty to make the removal its own undo step.
- Redo works after an Undo leaves the caret beside an empty verse or a
  trailing note. Caret-host commits (create, move, removal on blur) are
  tagged history-merge through `$tagCaretHostCommit`, so they no longer
  become an undo step that clears the redo stack.
- Drop the testUSFM-2SA-2/2SA-3 rows from both typing-oracle
  expected-failure lists: the Undo/Redo shapes they pinned now pass.
- Pin an author's literal `~` in a milestone attribute value turning into
  U+00A0 on a Standard-view settle with `it.fails` (needs a product call).
- Use a code span for `USJ_VERSION` in `Usj.version`'s TSDoc so
  consumers' typedoc can resolve the comment.
- Rebuild the committed dist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session-URL: https://claude.ai/code/session_018WCZ5GnNin2H8EzQyWWq6p

* fix(platform): keep a pasted no-break space at the start of a line

Standard view's paste read a U+00A0 at the start of the pasted text, or
of any line of it, as display spacing and inserted a plain space. So a
pasted lone no-break space became a space, one beside a space collapsed
away, and with allowInvisibleCharacters on the editor's own copy of a
stored no-break space pasted back as a space or as nothing (the copy
spells a stored no-break space as U+00A0 there).

No copy reaching the paste puts display spacing at a line start: the
editor's own copy writes a plain space for every display NBSP in both
readable flavors, a browser's default copy writes text/plain with every
NBSP as a space, and a Paratext 9 clipboard is decoded with its NBSPs
resolved. normalizePastedNbsp now keeps every pasted U+00A0 as a stored
no-break space except one right beside a marker token, which is still
read as a separator.

pastedNbsp.test.tsx pastes each case through the editor in both
settings: from a foreign text/plain and html-only clipboard, and the
editor's own copy pasted back. Two rows stay expected failures with the
setting on: a stored no-break space right after a closer or right
before an opener copies as U+00A0 and pastes back as a space under the
marker-adjacent rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(platform): stop exporting NBSP_STAND_IN_FONT_FAMILY

Hosts name the no-break-space stand-in's font family in their own
stylesheets, which cannot import a constant, so the export gave them
nothing to use. The constant stays internal to generateUsjCss and the
font registration, and the docs now state the family name,
PlatformEditorNbspStandIn, which must lead any host Scripture
font-family and must not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): keep an author's lone no-break space in delta ops

- The delta-op producer dropped every text that is exactly one U+00A0
  inside a note's contents, and every char span whose only text is one
  U+00A0 (as an empty span's placeholder). In views that hold a stored
  no-break space as U+00A0 itself (formatted, paragraph-structure), that
  is also a `~` between two of a note's char spans, or a `~` that is a
  span's whole text.
- Both checks now ask `$isLoneNbspContent`, the rule the editor-to-USJ
  conversion already keeps content by, so a note's ops (`getNoteOps`,
  which Simple mode's footnotes pane edits) carry what `getUsj()` saves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(platform): Standard view's delta ops read as USJ text in both no-break-space settings

- `opsForHost`/`opsFromHost` told a marker's separator from a stored
  no-break space only within one op, and took any op that is exactly one
  U+00A0 as a spacer. The editor emits each text node as its own op, so a
  glyph's separator (`\nd`, then ` A`) went out as a space, and with
  `allowInvisibleCharacters` on a stored no-break space that is a whole
  op's text (a `\ft` that is only `~`, a `~` between two spans or after a
  note, text carried out past a note) came back in as display spacing and
  was saved as a space or dropped. An NBSP is now structural only right
  after an opening glyph, in the same op or ending the op before.
- With `allowInvisibleCharacters` off, Standard view's ops went out as its
  display text: `~` for a stored no-break space and U+00A0 for each space
  of a run. Simple mode's footnotes pane edits a paragraph-structure
  editor's notes in Standard view, so a pane edit saved a space run as
  no-break spaces, and a stored no-break space showed in the pane as a
  blank rather than `~`. Both settings now map the same way: the stored
  no-break space's spelling to U+00A0 and a run's NBSPs to spaces going
  out, U+00A0 to the spelling coming in.
- A paragraph-structure editor still reads a char span whose whole text
  is one U+00A0 from applied ops as an empty span's placeholder, so the
  pane drops that `~`; pinned with `it.fails`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(platform): mark toUsfmNbspSpelling experimental

It stays exported, but is expected to be removed once Standard view
holds a stored no-break space as itself (PT-4931): display text will
then already carry the project's spelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): keep an applied no-break space that is a char span's whole text

- Applying delta ops built every char span's text without the mark a
  load puts on a span's no-break space (`authoredNbspState`). In the
  marker modes without a separator after a span's opener (formatted,
  paragraph-structure), a span whose whole text is one U+00A0 then read
  as an empty span's placeholder and saved as nothing: Simple mode's
  footnotes pane dropped a `\ft ~` (or any `~`-only span) its row editor
  handed back, and the same happened to a `~`-only span inserted by
  `applyUpdate`.
- The op producer leaves an empty span's placeholder out of its ops, so
  an applied U+00A0 is always content. `$markAppliedNbsp` marks the text
  of every char-span insert (`$handleCharText`, a note's contents, an
  unknown embed's contents) that holds one, as the USJ adaptor marks a
  loaded span's text; editable marker mode is left unmarked, as on a
  load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(platform): expect the footnotes pane to keep a `~`-only char span

The paragraph-structure main editor now marks an applied lone no-break
space as content, so the row that pinned the drop with `it.fails` runs
as an ordinary test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): keep a char span or milestone in text carried out past a note

Text a host's note editor carries out past its note (`takeOpsAfterNote`,
applied after the note with `replaceEmbedUpdate`) lost its markup:

- A char span from a Standard-view note editor became an unclosed span
  plus a stray `\nd*` in Standard view, and its glyphs became span text
  in the paragraph-structure and formatted views. The carried ops held
  the span's marker glyphs as text, the shape of the note editor's own
  delta document. `getOpsAfterNote`/`takeOpsAfterNote` now produce
  content-only ops (`$getParticularNodeOps`'s new `contentOnly` option),
  the shape a note's contents already have, and `takeOpsAfterNote`
  deletes what it carries by the nodes' apply-coordinates length
  (`$getApplyLength`), since the ops no longer count the glyphs.
- Applying a char span in editable marker mode added its glyphs but
  advanced past its text alone, so the ops after it landed inside the
  span's opener. `$handleCharText` now gives a fresh span the separator
  after its opener, as a load does, and counts everything it added.
- Plain text inserted at the end of a nested span went into the outer
  span; it now goes after the outermost span.
- A milestone vanished in every view: the op producer writes a milestone
  under `milestone`, and the applier read only `ms`. It now reads both.

Standard view still splits a carried `\nd L \+wj M\+wj*\nd*` into two
`\nd` spans, pinned with `it.fails` in noteCarriedText.test.tsx.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(platform): pin copying a whole note through the editor's own copy

A host's copy-this-note button selects the whole note, calls
EditorRef.copy() and puts the selection back with setSelection(). Pin
that the sequence writes the USFM a Ctrl+C of the note writes (no
display NBSP, a stored NBSP as ~ or U+00A0 by setting, no closer an
unclosed note lacks, no text after the note), applies the copy limit
(a note that does not fit copies nothing), and leaves the selection,
the document and a pending marker edit as they were when it runs in
one synchronous handler. The mount helper takes a copy limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(platform): read a pasted no-break space as a space only after an opening marker

A paste turned a U+00A0 right after any marker token, or right before
one, into a space. With allowInvisibleCharacters on, Standard view's own
copy writes a stored no-break space as U+00A0, so copying one that sat
after a closer or in front of a marker and pasting it back saved a
space.

Only a U+00A0 directly after an opening marker (\nd, \+nd) is now read
as a space: the marker's separator, as ParatextData reads it. After a
closer or a milestone's \* self-closer, and in front of any marker, a
U+00A0 is kept as a stored no-break space. A foreign copy of the display
that carries a run's or a note layout's spacing there (the⍽\nd,
\f +⍽\fr) now keeps it as a no-break space rather than a space: nothing
is lost. The two own-copy round-trip rows that pinned the old reading as
expected failures now pass; the paste suites are re-pinned to the new
reading, and a pasted \nd~Lord\nd* is pinned to keep every byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): emit milestone ops under the ms key the op types declare

`$getMilestoneOp` wrote a milestone as `{ insert: { milestone: … } }`, but
`OTEmbedTypes`/`validOTEmbedTypes` declare `ms`, and so do
`isInsertEmbedOpOfType` and the applier. A host reading ops through the
public type guard never saw a milestone, and `getInsertedNodeKey` (which
only recognises `validOTEmbedTypes`) could not resolve one. The producer
now writes `ms`; the applier keeps reading `milestone` too, so ops from
earlier builds still apply their milestones.

Tests that pinned `milestone` (adaptor ops, the shared converter
fixtures) now expect `ms`, the applier's `milestone` test is named as the
legacy-key case, and a produce-then-apply round trip pins the producer
key and that both milestones survive with sid/eid/who. Before the
producer change the round trip's apply half already passed (4e43000f
made the applier read both keys); the key pin failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): build adjacent char ops of different attributes as separate spans

Applying ops joined a char op into the span before it, or into the span
at the insertion point, when the two matched on marker and cid alone. So
the footnote editor's round trip (getNoteOps, applyUpdate,
replaceEmbedUpdate) turned `\w a|lemma="x"\w*\w b|lemma="y"\w*` into
`\w ab|lemma="x"\w*` in every view, and an op inserted right before a
`\w` span of another lemma went into that span's text, losing its own
lemma (and, for a nested char array, its outer span).

`$hasAllSameCharAttributes` (shared) compares the marker, the cid and
every other attribute. `$createNestedChars` and `$handleCharText` use it
to decide whether an op continues an existing span, and
`$charNodeTransform` uses it in place of its own attribute comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): keep adjacent spans that arrive in one paste apart

In the views that hide marker glyphs, `$charNodeTransform` joined two
alike spans that a paste inserted side by side, so pasting the editor's
own copy of `\w a\w*\w b\w*` or `\nd A\nd*\nd B\nd*` saved one span.
Standard view, and Paratext, keep them as the two spans the copied
content held.

`CharNodePlugin` records the char spans each clipboard insertion brings
in (`SELECTION_INSERT_CLIPBOARD_NODES_COMMAND`, and the protected-document
paste and drop, which insert directly), and `$charNodeTransform` keeps
two of them apart for the rest of that update. A pasted span still joins
an alike span the document held, so pasting a piece of a span inside a
span of the same marker extends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(platform): rebuild the committed dist for the merged paste and ops fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): count an author's `~`-only char span in delta-doc positions

- The ops stream emits a char span whose whole text is an author's
  no-break space (`$isLoneNbspContent`), but delta-doc counting still
  took every lone U+00A0 that is a span's only child as the empty span's
  placeholder and counted it 0. Every delta-doc position after such a
  span (`\nd ~\nd*` in formatted, paragraph-structure and the other views
  without a separator after the opener) was one short of the ops stream,
  so a fast-path keystroke after it went out with a retain one too small.
- The placeholder check now asks `$isLoneNbspContent`, as the producer
  does. Apply coordinates are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): keep a note's `~` that follows the opener's separator in delta ops

- In editable marker mode (unformatted) the text after a note span's
  opener is the separator followed by the author's text. For `\ft ~\ft*`
  or `\fk ~\+wj x\+wj*\fk*` that is two no-break spaces; the producer
  stripped the separator and then asked `$isLoneNbspContent` about the
  node, whose whole text is not one U+00A0, so the author's `~` was taken
  for layout and dropped from the note's ops. `getNoteOps` and
  `getEditorDelta` then disagreed with `getUsj()`, which keeps it.
- A lone NBSP left after stripping a separator is now content; the
  position rule is asked only about text that kept its node's every byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): give an applied `~` the separator after a span's opener in editable mode

- `$createNestedChars` told an empty span's placeholder from first
  content by its bytes, so an op whose text is one U+00A0 (an author's
  `~` as a span's whole text, or leading a nested span) got no separator
  after the opener in editable marker mode. Saving then stripped the `~`
  as the separator: in the unformatted view, the footnote popover's save
  and the Simple-mode pane round trip both lost `\ft ~\ft*` and
  `\fk ~\+wj x\+wj*\fk*`.
- The placeholder is now known by where it came from (an empty op's
  text), carried through the merge recursion; every op's own text takes
  the separator. An empty op still gets only the placeholder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(platform): pin that saved USJ and delta ops agree on every lone `~`

- `lone-nbsp-agreement.test.tsx` runs each lone-`~` shape (a span's whole
  text, between two spans, leading an element-first span; in body text
  and in a note) through every view in both settings of
  `allowInvisibleCharacters`: the saved USJ keeps it, `getEditorDelta`
  carries the same lone NBSPs, delta-doc positions after it match the
  ops stream, and a note's `~` survives a save of its own ops, the
  footnote popover's round trip and the footnotes pane's round trip.
  Text typed into an element-first spacer or an empty span's
  placeholder saves as content.
- The popover round-trip suite gains an authored `\fq ~\fq*` and
  `\fk ~\+wj x\+wj*\fk*` through the popover's load and save, in the
  editing views and unformatted.

Tests adapted from Jolie Rabideau's #36.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(shared-react): keep a newly marked span apart from an alike span beside it

In the views that hide marker glyphs, applying a character marker to the
text right after a span of the same marker joined the new span into the
old one: the marker menu or `insertMarker('nd')` on "God" after
`\nd Lord\nd*` saved `\nd LordGod\nd*`. Paratext 9 and Standard view make
a second span.

`getUsjMarkerAction` records every span a marker apply creates
(`$recordNewlyMarkedSpan`), and `$charNodeTransform` keeps a recorded span
apart from an alike span the document already held. The pieces one apply
creates still combine with each other; Enter-then-Backspace rejoining,
the op applier's nested fragments, a paste extending a span, and
`extendCharacterMarker` record nothing and keep combining.

A selection that reaches into part of an alike span and past it now
extends that span instead of nesting a copy inside it. This deliberately
departs from Paratext 9, which mangles the line in that case (TJ,
2026-10-27). Standard view kee…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants