feat(engines): font resolver, HarfBuzz shaper and Unicode tables for the layout engine

THIRD-PARTY-NOTICES.md

@@ -101,6 +101,12 @@ Windows headers, without a C runtime. Pieces of these projects that can end up i
 the x86 branch filter of the LZMA SDK (public domain, Igor Pavlov, https://www.7-zip.org/sdk.html), and the published
 app as its payload. Filee 1.x's installer was built with Inno Setup, which is no longer used.
 
+## Unicode data
+
+`src/Filee.Engines/Typesetting/UnicodeData.Generated.cs` holds Line_Break, Bidi_Class, Script and East_Asian_Width
+tables that `build/generate-unicode-tables.ps1` generates from the Unicode Character Database 16.0.0 (Unicode License
+v3, Copyright © 1991-2024 Unicode, Inc., https://www.unicode.org/license.txt).
+
 ## Fonts and icons
 
 | Component | License | Project |

build/generate-unicode-tables.ps1

@@ -0,0 +1,224 @@
+<#
+.SYNOPSIS
+  Generates the Unicode property tables of Filee's layout engine
+  (src/Filee.Engines/Typesetting/UnicodeData.Generated.cs) from the Unicode Character Database.
+
+.DESCRIPTION
+  Writes Line_Break (UAX #14), Bidi_Class (UAX #9), Script (UAX #24) and East_Asian_Width (UAX #11) as range tables
+  that Typesetting/UnicodeData.cs searches. Every code point is covered: unlisted ones get the defaults the files
+  declare in their "@missing" lines. Each enum lists the property's overall default first (XX, L, N, Unknown), so its
+  value 0 is what code points outside the code space get.
+
+  The version is pinned to UCD 16.0.0, the one .NET 10 uses for System.Globalization (CharUnicodeInfo, Rune,
+  StringInfo), so the layout engine and .NET agree on which characters exist and where grapheme clusters end. Move
+  both together: when the target framework changes its Unicode version, update $Version and the hashes below and run
+  the script again.
+
+  The UCD files are downloaded from unicode.org into build/.cache and verified with SHA-256; they are not committed.
+  The generated C# file is committed, so building Filee needs neither the script nor the network.
+
+.PARAMETER Output
+  The C# file to write. Default: src/Filee.Engines/Typesetting/UnicodeData.Generated.cs.
+
+.EXAMPLE
+  pwsh build/generate-unicode-tables.ps1
+#>
+[CmdletBinding()]
+param(
+    [string]$Output = (Join-Path $PSScriptRoot '..\src\Filee.Engines\Typesetting\UnicodeData.Generated.cs')
+)
+
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue' # Invoke-WebRequest is ~10x faster without the progress bar
+
+$Version = '16.0.0'
+# UCD file => SHA-256. Update together with $Version.
+$files = [ordered]@{
+    'LineBreak.txt'                  = 'E97E4259D0D20FAB150B9C7B4B28ABFAE5CD78CA97E7F4AC6ED20D685D5F4A7C'
+    'extracted/DerivedBidiClass.txt' = '71ED943A49C58568D8D92E80ECC2BA2F06E62AEE9C8EBB0E6E8BD2C3ED8B180E'
+    'Scripts.txt'                    = '9E88F0A677DF47311106340BE8EDE2ECDACD9C1C931831218D2BE6D5508E0039'
+    'EastAsianWidth.txt'             = '43ADC76C0686A42CB370764EB8CFE2B2A45B10B855E5572A2DB4A0EECCE15D5B'
+    'PropertyValueAliases.txt'       = '440FD3E5460B9BFE31DA67B6F923992E1989D31FE2ED91E091C4B8F8E2620BF9'
+}
+
+$cache = Join-Path $PSScriptRoot ".cache\ucd-$Version"
+New-Item -ItemType Directory -Force -Path $cache | Out-Null
+
+function Get-UcdFile([string]$name) {
+    $expected = $files[$name]
+    $file = Join-Path $cache ($name -replace '/', '_')
+    if ((Test-Path $file) -and (Get-FileHash $file -Algorithm SHA256).Hash -eq $expected) {
+        Write-Host "ok   $name (cached)"
+        return $file
+    }
+    $url = "https://www.unicode.org/Public/$Version/ucd/$name"
+    Write-Host "get  $url"
+    Invoke-WebRequest -Uri $url -OutFile $file -UseBasicParsing
+    $actual = (Get-FileHash $file -Algorithm SHA256).Hash
+    if ($actual -ne $expected) {
+        Remove-Item $file
+        throw "Checksum mismatch for $name. Expected $expected, got $actual."
+    }
+    return $file
+}
+
+# ───────────────────────── Property value names ─────────────────────────
+
+# short name => long name, per property, in the order PropertyValueAliases.txt lists them.
+$aliases = @{}
+foreach ($line in Get-Content (Get-UcdFile 'PropertyValueAliases.txt')) {
+    if ($line -match '^\s*(lb|bc|ea|sc)\s*;\s*([^;#\s]+)\s*;\s*([^;#\s]+)') {
+        if (-not $aliases.ContainsKey($Matches[1])) { $aliases[$Matches[1]] = [ordered]@{} }
+        $aliases[$Matches[1]][$Matches[2]] = $Matches[3]
+    }
+}
+
+<# Values of one property for every code point. The "@missing" defaults come first, in file order (later lines are
+   more specific), then the data lines. $names maps every name the file uses (short or long) to a value index. #>
+function Read-Property([string]$file, [hashtable]$names) {
+    $values = [byte[]]::new(0x110000)
+    $missing = [System.Collections.Generic.List[object]]::new()
+    $data = [System.Collections.Generic.List[object]]::new()
+    foreach ($line in Get-Content $file) {
+        if ($line -match '^#\s*@missing:\s*([0-9A-F]+)\.\.([0-9A-F]+)\s*;\s*([^;#\s]+)') {
+            $missing.Add(@([Convert]::ToInt32($Matches[1], 16), [Convert]::ToInt32($Matches[2], 16), $Matches[3]))
+        }
+        elseif ($line -match '^([0-9A-F]+)(?:\.\.([0-9A-F]+))?\s*;\s*([^;#\s]+)') {
+            $first = [Convert]::ToInt32($Matches[1], 16)
+            $last = if ($Matches[2]) { [Convert]::ToInt32($Matches[2], 16) } else { $first }
+            $data.Add(@($first, $last, $Matches[3]))
+        }
+    }
+    if ($missing.Count -eq 0) { throw "$file has no @missing line." }
+    foreach ($range in @($missing) + @($data)) {
+        $name = $range[2]
+        if (-not $names.ContainsKey($name)) { throw "$file uses the unknown value '$name'." }
+        [Array]::Fill[byte]($values, [byte]$names[$name], $range[0], $range[1] - $range[0] + 1)
+    }
+    return , $values
+}
+
+<# The values as ranges: (start, value) where the value changes. #>
+function Get-Ranges([byte[]]$values) {
+    $starts = [System.Collections.Generic.List[int]]::new()
+    $result = [System.Collections.Generic.List[byte]]::new()
+    $previous = -1
+    for ($i = 0; $i -lt $values.Length; $i++) {
+        if ($values[$i] -ne $previous) {
+            $starts.Add($i)
+            $result.Add($values[$i])
+            $previous = $values[$i]
+        }
+    }
+    return , @($starts, $result)
+}
+
+<# A C# enum member name from a UCD long name: "Caucasian_Albanian" => "CaucasianAlbanian". #>
+function ConvertTo-Member([string]$name) {
+    return (($name -split '_') | ForEach-Object { $_.Substring(0, 1).ToUpperInvariant() + $_.Substring(1) }) -join ''
+}
+
+# ───────────────────────── Writing ─────────────────────────
+
+$out = [System.Text.StringBuilder]::new()
+function Line([string]$text = '') { [void]$out.Append($text).Append("`n") }
+
+<# One property: the enum (members in $order, as short or long names) and its range table. #>
+function Write-Property([string]$enum, [string]$summary, [string]$table, [string]$file, [string]$property,
+    [string[]]$order, [switch]$LongMembers) {
+    $names = @{}
+    $long = $aliases[$property]
+    for ($i = 0; $i -lt $order.Count; $i++) {
+        $short = $order[$i]
+        $names[$short] = $i
+        $names[$long[$short]] = $i
+    }
+    $values = Read-Property $file $names
+    $ranges = Get-Ranges $values
+    Write-Host ("{0,-16} {1,3} values, {2,5} ranges" -f $enum, $order.Count, $ranges[0].Count)
+
+    Line "/// <summary>$summary</summary>"
+    Line "internal enum $enum : byte"
+    Line '{'
+    for ($i = 0; $i -lt $order.Count; $i++) {
+        $short = $order[$i]
+        if ($LongMembers) {
+            Line "    /// <summary>$short</summary>"
+            Line "    $(ConvertTo-Member $long[$short]),"
+        }
+        else {
+            Line "    /// <summary>$($long[$short] -replace '_', ' ')</summary>"
+            Line "    $short,"
+        }
+        if ($i -lt $order.Count - 1) { Line }
+    }
+    Line '}'
+    Line
+
+    $script:tables += , @($table, $enum, $ranges)
+}
+
+$script:tables = @()
+$lb = $aliases['lb'].Keys | Where-Object { $_ -ne 'XX' }
+$bc = $aliases['bc'].Keys | Where-Object { $_ -ne 'L' }
+$ea = $aliases['ea'].Keys | Where-Object { $_ -ne 'N' }
+$sc = $aliases['sc'].Keys | Where-Object { $_ -notin 'Zzzz', 'Zyyy', 'Zinh' }
+
+Line '// <auto-generated>'
+Line "// Generated by build/generate-unicode-tables.ps1 from the Unicode Character Database $Version (LineBreak.txt,"
+Line '// DerivedBidiClass.txt, Scripts.txt, EastAsianWidth.txt and PropertyValueAliases.txt). Do not edit: run the'
+Line '// script. The data is Copyright (c) Unicode, Inc., under the Unicode License v3 (see THIRD-PARTY-NOTICES.md).'
+Line '// </auto-generated>'
+Line
+Line 'namespace Filee.Engines.Typesetting;'
+Line
+
+Write-Property 'LineBreakClass' 'Line_Break property values (UAX #14), by their short names.' 'LineBreak' `
+(Get-UcdFile 'LineBreak.txt') 'lb' (@('XX') + $lb)
+Write-Property 'BidiClass' 'Bidi_Class property values (UAX #9), by their short names.' 'Bidi' `
+(Get-UcdFile 'extracted/DerivedBidiClass.txt') 'bc' (@('L') + $bc)
+Write-Property 'EastAsianWidth' 'East_Asian_Width property values (UAX #11).' 'EastAsianWidth' `
+(Get-UcdFile 'EastAsianWidth.txt') 'ea' (@('N') + $ea) -LongMembers
+Write-Property 'UnicodeScript' 'Script property values (UAX #24); the summary is the ISO 15924 code.' 'Script' `
+(Get-UcdFile 'Scripts.txt') 'sc' (@('Zzzz', 'Zyyy', 'Zinh') + $sc) -LongMembers
+
+Line 'internal static partial class UnicodeData'
+Line '{'
+Line '    /// <summary>Version of the Unicode Character Database the tables were generated from.</summary>'
+Line "    public const string Version = `"$Version`";"
+Line
+Line '    /// <summary>ISO 15924 code of each <see cref="UnicodeScript"/>, by its value.</summary>'
+$codes = @(@('Zzzz', 'Zyyy', 'Zinh') + $sc | ForEach-Object { "`"$_`"" })
+Line '    private static readonly string[] ScriptCodes ='
+Line '    ['
+for ($i = 0; $i -lt $codes.Count; $i += 14) {
+    Line ('        ' + ($codes[$i..([Math]::Min($i + 13, $codes.Count - 1))] -join ', ') + ',')
+}
+Line '    ];'
+foreach ($entry in $script:tables) {
+    $table, $enum, $ranges = $entry
+    $starts, $values = $ranges
+    Line
+    Line "    /// <summary>First code point of each range of equal <see cref=`"$enum`"/> values, ascending.</summary>"
+    Line "    private static ReadOnlySpan<int> ${table}Starts =>"
+    Line '    ['
+    for ($i = 0; $i -lt $starts.Count; $i += 10) {
+        $count = [Math]::Min(10, $starts.Count - $i)
+        Line ('        ' + (($starts.GetRange($i, $count) | ForEach-Object { '0x{0:X4}' -f $_ }) -join ', ') + ',')
+    }
+    Line '    ];'
+    Line
+    Line "    /// <summary>The <see cref=`"$enum`"/> value of each range in <see cref=`"${table}Starts`"/>.</summary>"
+    Line "    private static ReadOnlySpan<byte> ${table}Values =>"
+    Line '    ['
+    for ($i = 0; $i -lt $values.Count; $i += 24) {
+        $count = [Math]::Min(24, $values.Count - $i)
+        Line ('        ' + (($values.GetRange($i, $count) | ForEach-Object { "$_" }) -join ', ') + ',')
+    }
+    Line '    ];'
+}
+Line '}'
+
+$Output = [IO.Path]::GetFullPath($Output)
+[IO.File]::WriteAllText($Output, $out.ToString(), [Text.UTF8Encoding]::new($false))
+Write-Host "Wrote $Output"

docs/ENGINES.md

@@ -602,12 +602,15 @@ document). If tagged PDF (PDF/UA) becomes a goal, it would need post-processing
 
 ### Still open
 
-- Variable fonts: the subset is the default instance. A non-default instance (a weight the font resolver picks from a
-  variable font) needs an instancer that applies gvar / HVAR deltas before subsetting; until then such a typeface
-  would go to Skia whole (Type 3).
-- CFF2 outlines can't be converted or subset yet (Type 3 through Skia).
+- Variable fonts: the subset is the default instance. A non-default instance needs an instancer that applies gvar /
+  HVAR deltas before shaping and subsetting. The font resolver doesn't pick one: it uses a variable font at its default
+  instance and reports a missing weight as synthetic (see [fonts, shaping and Unicode
+  tables](#filee-renderer-fonts-shaping-and-unicode-tables)).
+- CFF2 outlines can't be converted or subset yet (Type 3 through Skia). The font resolver uses a CFF2 font only when no
+  other font for its script is installed.
 - HarfBuzzSharp's `Blob.FromStream` let HarfBuzz read freed memory in the spike's shaper (wrong glyphs, then a
-  crash); the shaper should hand HarfBuzz font data in memory it owns, as `Vector/GlyphOutlines.cs` does.
+  crash). The layout engine's shaper (`Typesetting/Shaper.cs`) hands HarfBuzz `PageFont.Data` pinned for as long as
+  HarfBuzz's face lives.
 
 The spike also fixed CID-keyed CFF fonts whose Top DICT and font dictionary both carry a units-per-em scale (Top DICT
 1/2048, font dictionary 1/1000, as in Pretendard): the two FontMatrix scales were multiplied, so OTF → TTF and EOT gave
@@ -619,6 +622,68 @@ trips); `tests/Filee.Engines.Tests/PdfPageWriterTests.cs` builds the page above
 PDFium picture, with each installed Korean font (Malgun Gothic, Noto Sans CJK, Noto Sans KR VF; skipped when none is
 installed), plus page sizes, links and the outline.
 
+## Filee renderer: fonts, shaping and Unicode tables
+
+The text services of Filee's own document renderer (GitHub Project 4, item *Layout engine: font resolver, HarfBuzz
+shaper and Unicode tables*), in `src/Filee.Engines/Typesetting`. No converter uses them yet; paragraph layout, CJK
+typography and the bidirectional algorithm build on them. They use only what Filee already runs in-process: SkiaSharp's
+font manager, HarfBuzzSharp and .NET.
+
+- **Font slots** (`FontSlots.cs`): which of a run's fonts draws a character. 한글's seven language slots (한글, 영문, 한자,
+  일어, 외국어, 기호, 사용자) go by script: ASCII, Latin-1 and the spacing modifiers to 영문, quotes, dashes, CJK
+  punctuation, fullwidth forms and other symbols to 기호, private use to 사용자, Greek, Cyrillic, Arabic, Thai, ... to
+  외국어. Word's four run fonts: ASCII to `ascii`, right-to-left, Brahmic and Southeast Asian scripts to `cs`, CJK
+  scripts and blocks to `eastAsia`, the rest to `hAnsi`; with `w:hint="eastAsia"` ambiguous-width characters and
+  private use go to `eastAsia`. Word's finer rule for Latin-1 letters in Chinese text (hint plus East Asian language)
+  isn't made. A grapheme cluster takes the slot of its first character, so marks stay with their base.
+- **Font resolver** (`FontResolver.cs`, `FontNames.cs`, `SystemFonts.cs`): installed fonts through `SKFontManager`
+  (DirectWrite, CoreText, fontconfig); nothing is downloaded or shipped.
+  - Names: a family is found by its name, by its name without trailing style words ("Pretendard Medium" is weight
+    500), and by other names: Korean ↔ English names (함초롬바탕 ↔ HCR Batang, 맑은 고딕 ↔ Malgun Gothic, the HY
+    fonts, Nanum fonts; checked against the fonts' name tables), the fonts built into 한글 (한양신명조, 한양중고딕, ...)
+    → the HY TrueType fonts of the same design, and metric-compatible replacements (Arial → Liberation Sans, Calibri →
+    Carlito, ...). Generic families (serif, sans-serif, monospace) get the language's default fonts.
+  - Style: the face nearest the requested weight and slant by the CSS matching rules. A bold or italic the family
+    lacks is reported as synthetic (`ResolvedFont.SyntheticBold` / `SyntheticItalic`) for the drawing to fake.
+  - Missing fonts and the document-wide rules come from `RhwpFonts.cs`, now shared (`FontSubstitution.cs`): a missing
+    font is replaced by an installed one of the script its name points to (else the text's language) and the same
+    style (serif or sans); `Plan` swaps Korean default fonts in Chinese or Japanese text (#64) and gives text without
+    Hangul, Han or kana one font that has all of it. rhwp keeps using the same rules through `RhwpFonts`.
+  - Per-cluster fallback: a run is split into pieces each drawn by one font. A grapheme cluster keeps the run's font
+    when it has every character of it, else takes a font already used in the run, a font for its script (a list per
+    script: Nirmala UI, Leelawadee UI, Noto, Lohit, ... ; Han by the text's language), then the platform's own
+    fallback (`SKFontManager.MatchCharacter`). Punctuation and spaces stay with the font of the text around them.
+  - Variable fonts are used at their default instance: HarfBuzzSharp 8.3 has no binding for variation coordinates, the
+    PDF subset is the default instance, and Filee has no instancer. DirectWrite's named instances of one variable
+    font count as one face, with the weight the font's own OS/2 table gives; a heavier weight is synthetic bold.
+  - CFF2 fonts (Source Han VF and the like) are used only when no other font for their script is installed: the
+    subsetter can't take them, so Skia would embed them as Type 3 fonts (still searchable, not subset).
+- **Shaper** (`Shaper.cs`): HarfBuzz shaping generalised from `Vector/GlyphOutlines.cs`. A run is shaped with up to
+  five characters of the paragraph on either side as context (HarfBuzz's own context length), so a word split into two
+  runs still joins and reorders as one. Results are in font units, cached by font, features, script, direction,
+  language and the run's text with its context, and carry glyph IDs, advances, offsets, HarfBuzz's cluster per glyph
+  (monotone graphemes, visual order) and its unsafe-to-break flags for the line breaker. `ShapedRun.ToGlyphRun` makes
+  the page model's `GlyphRun` (`PdfPageWriterTests` now builds its page with it). `Shaper.Metrics` gives ascender,
+  descender, line gap, Win metrics, x-height, cap height, underline, strikeout and super / subscript metrics.
+- **Unicode tables** (`UnicodeData.cs`, `UnicodeData.Generated.cs`): Line_Break, Bidi_Class, Script and
+  East_Asian_Width for every code point, as range tables searched by binary search; unassigned code points get the
+  UCD's defaults (`@missing` lines: ID for the CJK blocks, R and AL for the right-to-left blocks, ...).
+  `build/generate-unicode-tables.ps1` downloads the pinned UCD files from unicode.org into `build/.cache` (SHA-256
+  checked, not committed) and writes the C# file, which is committed (Unicode License v3, in THIRD-PARTY-NOTICES.md).
+  - Version: **Unicode 16.0.0**, the version .NET 10 uses for `CharUnicodeInfo`, `Rune` and `StringInfo` (checked by
+    a test over every code point), so the layout engine's properties and .NET's grapheme clusters and categories agree.
+    HarfBuzz 8.3 inside HarfBuzzSharp uses Unicode 15.1 for its own normalization and mark handling; the scripts added
+    in 16.0 (Garay, Todhri, Sunuwar, ...) shape with HarfBuzz's default shaper until HarfBuzzSharp moves on. Change the
+    version together with the target framework: update `$Version` and the hashes and run the script.
+
+Tests: `FontSlotsTests`, `UnicodeDataTests`, `FontResolverTests` (a fake font list like `RhwpFontsTests`, plus one
+check of the installed Malgun Gothic) and `ShaperTests`. The shaper is checked three ways for Arabic, Devanagari,
+Gujarati, Thai and Hangul jamo: against HarfBuzz itself (the text shaped the plain way, one buffer with guessed
+properties) with every installed font of the script, against recorded HarfBuzz output for the pinned Noto fonts of
+`build/fetch-fonts.ps1` (CI fetches them; there is no pinned Gujarati font), and by what each script needs whatever the
+font (joining forms and lam-alef, vowel signs drawn before their consonant, conjuncts, the SARA AM split, jamo composed
+into precomposed syllables, a dotted circle only for a broken cluster).
+
 ## DOCX writer
 
 The counterpart of the HWPX writer: `Hwp/Hwpx/Docx/DocxWriter*.cs` writes WordprocessingML by hand from the same