Skip to content

Control characters in run text are deleted on save, so Segment.BeginOffset and run boundaries shift after reload #415

Open

Description

@johnml1135

Characters XML cannot store (U+0000–U+0008, U+000B, U+000C, U+000E–U+001F, U+FFFE, U+FFFF) are accepted by the model, but TsStringSerializer silently deletes them on write (StripInvalidXmlChars). After reload, anything that stores a character index into the text points at the wrong character.

Repro (master 0bc98fb, net462 and net8.0):

  • Take a paragraph One a<U+0001>b two. Three four. whose second segment has BeginOffset 13 ("Three"). Save, close and reopen the project. The text is now 23 characters instead of 24, and BeginOffset 13 points at "hree ". The next parse or edit misplaces analyses.
  • The same happens with a plain string: runs ab<U+0001>cd (en) + EFGH (es) serialize and deserialize with the es run starting at 4 instead of 5.

ktptOffset is not affected: it is a baseline shift in millipoints, not a character index.

Suggested fix: make writes length-preserving by replacing these characters with U+FFFD instead of deleting them, and apply the same replacement at input in TsStringUtils.NormalizeNfd(ITsString). This is the approach #414 takes for lone surrogates. Whether to replace them or reject them at input is a maintainer decision. Failing tests are ready (ControlCharacterOffsetDesyncTests, and a TsStringSerializerTests round-trip) and can come with the fix PR.

Related: #409, #412, #414.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions