Skip to content

Use TextEncoder/TextDecoder for UTF-8 - #55

Merged
matmen merged 1 commit into
masterfrom
text-decoder
Oct 15, 2026
Merged

matmen merged 1 commit into
masterfrom
text-decoder

Conversation

@matmen

@matmen matmen commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

The library had its own UTF-8 encoder and decoder, meant for runtimes without TextEncoder/TextDecoder. Every supported runtime (Node 22.13+, Deno, Bun, browsers, workers) has both, so it now uses them.

  • More correct: the old decoder accepted overlong sequences and encoded surrogates, which could put lone surrogates into strings. TextDecoder replaces them with U+FFFD, as the encoding standard specifies. Valid UTF-8 decodes the same as before, including skipping a leading byte order mark.
  • Faster: the old decoder built strings one character at a time, which mattered for large SVGs.
  • Less code: src/util/utf8.ts goes from 59 to 16 lines.

The tsconfig only includes the es2022 lib, so the file declares the two classes itself instead of pulling in lib.dom. The declaration stays inside that module and isn't part of the published types.

Used for PNG text chunks and SVG input.

Every supported runtime has them. The hand-written decoder accepted
overlong sequences and encoded surrogates, which TextDecoder replaces with
U+FFFD as the encoding standard specifies, and it's faster on large SVGs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ljnxm1Qjrocq1izGajLx8J
@matmen matmen self-assigned this Oct 15, 2026
@matmen matmen added the enhancement New feature or request label Oct 15, 2026
@matmen
matmen merged commit bbeab78 into master Oct 15, 2026
13 checks passed
@matmen
matmen deleted the text-decoder branch October 15, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant