diff --git a/CHANGELOG.md b/CHANGELOG.md index 56864a4c4..1211ed493 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -316,6 +316,8 @@ pt-BR: Corrigidos texto chinês lido como coreano, símbolos do Word que sumiam, PowerPoint → PDF or HWPX as black dots (issue #350). They're the right characters now. - On Linux without Korean fonts, DOCX, XLSX and other documents converted to PDF through rhwp came out without their text (by @Perricheno). +- Some OpenType fonts with CFF outlines (Pretendard, for example) converted to TTF or EOT with every glyph shrunk to a + dot. They keep their size now. - Spreadsheets converted to PDF through LibreOffice could get completely empty pages. They're removed, with a note; PDF/A results are left as LibreOffice wrote them (by @Perricheno). - A file with a file name token in its own name had it filled in: `Minutes {date}.png` became `Minutes 20261022.jpg` diff --git a/docs/ENGINES.md b/docs/ENGINES.md index c76957ad7..219ac9504 100644 --- a/docs/ENGINES.md +++ b/docs/ENGINES.md @@ -500,6 +500,8 @@ extension, and the file must then have an NE header with FNT resources. supported. FON cannot be an output format. - Not supported: CFF2 (variable) outlines to TTF or EOT (to WOFF, WOFF2 and OTF they convert), writing font collections, and WOFF/WOFF2 extended metadata (dropped). +- Subsetting (`Fonts/FontSubsetter.cs`) is not a conversion: it serves the Filee renderer's PDF output, see + [Filee renderer: PDF output](#filee-renderer-pdf-output). `tests/Filee.Engines.Tests/FontTests.cs` checks every conversion with small OFL fonts in `tests/Filee.Engines.Tests/Fonts` (see `OFL.txt` there): round trips keep every table byte for byte, a WOFF2 made @@ -507,6 +509,110 @@ by Google's encoder decodes to the source font, and converted CFF glyphs are com `tests/Filee.Engines.Tests/FonTests.cs` builds synthetic FON samples in code (`FonBuilder`) and compares every rendered glyph with its source bitmap. +## Filee renderer: PDF output + +Groundwork for Filee's own document renderer (GitHub Project 4, "Filee: built-in engines", item *Layout engine: PDF +backend and font subsetting spike*). No converter uses it yet; the layout engine's page model will be drawn through it. + +**Recommended decision (for the maintainer to confirm, "Decisions to make" #1): write PDFs with Skia's PDF backend +(`SKDocument.CreatePdf`) and fonts subset by Filee (option A), not with a page writer of Filee's own on PDFsharp's +low-level objects (option B).** + +### How option A writes a PDF + +- `Rendering/PageContent.cs` records a page: shaped glyph runs (`GlyphRun`: glyph IDs, positions, the source text + and HarfBuzz's cluster per glyph), other drawing as `SKCanvas` calls, URL links, links to named destinations and the + destinations. The same record draws onto any `SKCanvas`, so PNG output comes from the same code as the PDF. +- `Rendering/PdfPageWriter.cs` collects the glyphs every page uses per font, subsets each font once for the + document, and draws the pages with the subset typefaces into `SKDocument.CreatePdf`. Each run is one text blob with + its UTF-8 text and clusters (`SKPositionedTextRunBuffer`): Skia writes a Type 0 / CIDFontType2 font with a subset + tag, a ToUnicode map built from the subset's cmap, and `/ActualText` for clusters that don't map one glyph to one + character (ligatures, Indic conjuncts and reordered vowel signs, Arabic contextual forms). Right-to-left runs are + drawn in logical order (glyphs reversed, each at its own position), so text comes out the way it was written in + readers that take it in content order. +- Afterwards PDFsharp adds what Skia can't: the outline (bookmarks, with `/XYZ` destinations) and exact page sizes. + Skia rounds the media box to whole points (A4 becomes 595 × 842) and draws from its top edge; the media box is set + to end at that edge (`[0 0.11 595.28 842]`), so nothing moves. Pages of whole-point sizes without an outline skip + this pass. +- `Fonts/FontSubsetter.cs` keeps the outlines of the used glyphs (and the components of composite glyphs) at their + original glyph IDs and empties the others; the glyph count is cut to the highest used ID + 1. Kept: glyf / loca, + head, hhea / hmtx, maxp, OS/2, a cmap of the used glyphs (`Fonts/CmapTable.cs`, formats 4 and 12), post 3.0, name + records 0–6 and the hinting tables (cvt, fpgm, prep, gasp: some CJK fonts build glyphs with instructions). Layout, + vertical, bitmap and variation tables are dropped. CFF fonts are converted to TrueType first, only the used glyphs + (`CffToTrueType`), so Skia always gets a TrueType font. Keeping glyph IDs costs about 1 KB per font after deflate + (loca and hmtx of the empty glyphs); hinting about 3 KB per font. +- Fonts the subsetter can't take (CFF2, colour fonts) are handed to Skia whole: it draws them as Type 3 fonts (still + searchable), as it does today. + +### What was measured + +One A4 page built in code (`PdfPageWriterTests`): Korean (11 pt, 6 lines), English with ligatures and quotes, Arabic +with harakat (right-to-left, right-aligned) and Hindi with conjuncts and reordered vowel signs, four headings with +rules, a link, an outline and a filled box. Latin and Arabic in Segoe UI, Devanagari in Nirmala UI, Korean in the font +of each row. Option B was a prototype on PDFsharp's low-level objects (Type 0 / CIDFontType2, Identity-H, +CIDToGIDMap Identity, its own ToUnicode CMap and per-cluster `/ActualText`, the same subsetter); it drew only text, so +its page had no rules or box. The prototype is not in the repository. + +| Korean font | A: PDF | A: fonts embedded | B: PDF | B: fonts embedded | +|---|---|---|---|---| +| Malgun Gothic (TrueType, 13.5 MB) | 72.7 KB | 28.1 + 19.3 + 12.4 KB | 73.4 KB | 27.8 + 19.0 + 12.2 KB | +| Noto Sans KR VF (variable TrueType, 10.4 MB) | 57.5 KB | 12.3 + 19.3 + 12.4 KB | 57.9 KB | 12.1 + 19.0 + 12.2 KB | +| Pretendard (CID-keyed CFF, 1.6 MB) | 57.2 KB | 12.4 + 19.3 + 12.4 KB | 57.7 KB | 12.2 + 19.0 + 12.2 KB | +| S-Core Dream (CFF, 0.7 MB) | 56.0 KB | 11.0 + 19.3 + 12.4 KB | 56.2 KB | 10.9 + 19.0 + 12.2 KB | + +- **Without Filee's subsetter**, Skia embeds the whole font: one line of Malgun Gothic makes a 7.45 MB PDF, one line of + Segoe UI 550 KB, and the variable fonts and CFF fonts above come out as Type 3 fonts. +- **Size**: both options under 200 KB with every font embedded as a subset (`ABCDEF+` tag, FontFile2), under 75 KB + in fact. The writers' own overhead differs by less than 1 KB. +- **Text**: PdfPig reads back the exact source text from both (content order, `/ActualText` spans replacing their + glyphs): every cluster whole, Arabic in logical order, with every Korean font. +- **Look in PDFium** (PDFtoImage, rendered at 300 dpi and averaged down to 100 dpi, compared with Skia's own picture + of the page; SSIM over the windows with ink): A 0.946–0.954 with 0.45–0.5 % of the pixels more than 64 grey levels + apart, B 0.960–0.971 with 0.07–0.2 %. B also matches PDFium's picture of the same glyphs drawn as paths (0.99), + A less (0.945). Crops at 300 dpi look the same: the difference is anti-aliasing, most likely because PDFium draws + Skia's text through a flipped text matrix (`1 0 0 -1`), not glyph shapes or positions (PdfPig finds all 966 glyphs + of both files within 0.001 pt of each other). +- **Speed**: A 0.17–0.4 s for the page including subsetting (1.1 s for Pretendard, whose 1.6 MB CFF is parsed whole), + B 0.03–0.17 s. +- **Not checked**: browsers (Chrome and Edge use PDFium, the engine checked; Firefox's pdf.js wasn't run), Adobe + Acrobat and macOS Preview; the full Noto Sans CJK OTF (not installed here: CID-keyed CFF is covered by + `FileeTest-CID.otf`, a subset of Noto Sans KR OTF, and by Pretendard locally); Linux and macOS. Set + `FILEE_RENDER_OUT` to a folder when running `PdfPageWriterTests` to keep the PDFs for a look in those viewers. + +### Why A + +- Text is a draw, not the hard part: both options write the same small, searchable PDFs, because the subsetter, the + cluster handling and the logical order do the work in both. +- Everything else a page has, A gets from Skia: paths, strokes and dashes, gradients, clipping, transparency, pictures + (JPEG passed through, others deflated with soft masks), links and destinations. B would need its own writer for each + of them, and the PNG route would need a second renderer that has to look the same (the planned check is SSIM ≥ 0.99 + between PNG and PDF). The EMF/WMF renderer item draws to `SKCanvas` and gets PDF for free with A. +- A's workarounds are small and contained in `PdfPageWriter`: exact page sizes and the outline through PDFsharp, a + document-wide glyph collection before drawing (the page model is complete by then). + +What A can't do, as of SkiaSharp 3.119: tagged PDF (Skia's structure tree API isn't in SkiaSharp), ActualText spans +other than per cluster, and control over the font dictionaries (Skia's subset tags are `AAAAAA+`, `BAAAAA+` … in every +document). If tagged PDF (PDF/UA) becomes a goal, it would need post-processing or option B. + +### Still open + +- Variable fonts: the subset is the default instance. A non-default instance (a weight the font resolver picks from a + variable font) needs an instancer that applies gvar / HVAR deltas before subsetting; until then such a typeface + would go to Skia whole (Type 3). +- CFF2 outlines can't be converted or subset yet (Type 3 through Skia). +- HarfBuzzSharp's `Blob.FromStream` let HarfBuzz read freed memory in the spike's shaper (wrong glyphs, then a + crash); the shaper should hand HarfBuzz font data in memory it owns, as `Vector/GlyphOutlines.cs` does. + +The spike also fixed CID-keyed CFF fonts whose Top DICT and font dictionary both carry a units-per-em scale (Top DICT +1/2048, font dictionary 1/1000, as in Pretendard): the two FontMatrix scales were multiplied, so OTF → TTF and EOT gave +glyphs a thousand times too small. They are now combined the way rasterizers do (`CffFont.Combine`). + +`tests/Filee.Engines.Tests/FontSubsetterTests.cs` checks the subsetter with the small OFL test fonts (same outlines +and advances at the same glyph IDs in Skia, composite components kept, CFF converted, tables dropped, cmap round +trips); `tests/Filee.Engines.Tests/PdfPageWriterTests.cs` builds the page above and checks size, fonts, text and the +PDFium picture, with each installed Korean font (Malgun Gothic, Noto Sans CJK, Noto Sans KR VF; skipped when none is +installed), plus page sizes, links and the outline. + ## DOCX writer The counterpart of the HWPX writer: `Hwp/Hwpx/Docx/DocxWriter*.cs` writes WordprocessingML by hand from the same diff --git a/src/Filee.Engines/Fonts/Cff/CffFont.cs b/src/Filee.Engines/Fonts/Cff/CffFont.cs index b63af2939..0d5e5ac88 100644 --- a/src/Filee.Engines/Fonts/Cff/CffFont.cs +++ b/src/Filee.Engines/Fonts/Cff/CffFont.cs @@ -114,8 +114,7 @@ public static CffFont Parse(byte[] data) var fdAt = Offset(fdArrayOffset[0], data); var fontDicts = ReadIndex(data, ref fdAt).Select(ParseDict).ToArray(); privates = [.. fontDicts.Select(fd => ReadPrivate(data, fd))]; - // A font dictionary's FontMatrix applies first, then the Top DICT's. - matrices = [.. fontDicts.Select(fd => fd.TryGetValue(OpFontMatrix, out var fm) && fm.Length == 6 ? Multiply(fm, topMatrix) : topMatrix)]; + matrices = [.. fontDicts.Select(fd => fd.TryGetValue(OpFontMatrix, out var fm) && fm.Length == 6 ? Combine(fm, topMatrix) : topMatrix)]; if (top.TryGetValue(OpFdSelect, out var fdSelectOffset)) fdSelect = ReadFdSelect(data, Offset(fdSelectOffset[0], data), charStrings.Length, privates.Length); } @@ -138,6 +137,27 @@ public static CffFont Parse(byte[] data) }; } + /// + /// The FontMatrix of a CID font's font dictionary combined with the Top DICT's: the font dictionary's applies + /// first, and the product is scaled back by the smaller of the two units-per-em (1 / scale), as rasterizers do. + /// Both matrices often carry a units-per-em scale of their own (Top DICT 1/2048 and font dictionary 1/1000 in + /// fonts such as Pretendard), and a plain product would shrink the glyphs a thousandfold. + /// + internal static double[] Combine(double[] fontDict, double[] top) + { + var product = Multiply(fontDict, top); + var topUnits = UnitsPerEm(top); + var fontDictUnits = UnitsPerEm(fontDict); + var scale = topUnits > 1 && fontDictUnits > 1 ? Math.Min(topUnits, fontDictUnits) : 1; + return [.. product.Select(v => v * scale)]; + + static double UnitsPerEm(double[] matrix) + { + var size = Math.Abs(matrix[3]) > 0 ? Math.Abs(matrix[3]) : Math.Abs(matrix[0]); + return size > 0 ? Math.Round(1 / size) : 1; + } + } + /// Combines two FontMatrix transforms: is applied, then . private static double[] Multiply(double[] first, double[] then) => [ diff --git a/src/Filee.Engines/Fonts/CffToTrueType.cs b/src/Filee.Engines/Fonts/CffToTrueType.cs index f89d1b0d7..03f60e3ac 100644 --- a/src/Filee.Engines/Fonts/CffToTrueType.cs +++ b/src/Filee.Engines/Fonts/CffToTrueType.cs @@ -24,8 +24,13 @@ internal static class CffToTrueType private static readonly byte[] SmoothGasp = [0, 1, 0, 1, 0xFF, 0xFF, 0, 0x0F]; /// Returns a TrueType-outline copy of . + /// + /// Only these glyphs are converted and the others are left empty (what the font subsetter needs); null converts + /// every glyph. + /// /// CFF2 (variable) outlines. - public static SfntFont Convert(SfntFont source, CancellationToken cancellationToken = default, IProgress? progress = null) + public static SfntFont Convert(SfntFont source, CancellationToken cancellationToken = default, IProgress? progress = null, + IReadOnlySet? only = null) { if (source["CFF2"] is not null) throw new NotSupportedException( @@ -49,6 +54,8 @@ public static SfntFont Convert(SfntFont source, CancellationToken cancellationTo cancellationToken.ThrowIfCancellationRequested(); progress?.Report((double)gid / glyphCount); } + if (only is not null && !only.Contains(gid)) + continue; var contours = interpreter.Run(gid); if (matrices[Math.Min(cff.FontDictIndex(gid), matrices.Length - 1)] is { } matrix) contours = [.. contours.Select(c => Transform(c, matrix))]; diff --git a/src/Filee.Engines/Fonts/CmapTable.cs b/src/Filee.Engines/Fonts/CmapTable.cs new file mode 100644 index 000000000..cf41c4e8f --- /dev/null +++ b/src/Filee.Engines/Fonts/CmapTable.cs @@ -0,0 +1,239 @@ +// The 'cmap' table (character → glyph): reads the font's best Unicode subtable (formats 4, 6, 10, 12, 13; a symbol +// font's (3,0) subtable when it has no Unicode one) and writes a new table from a code point → glyph map, as format 4 +// for the BMP plus format 12 when the map reaches beyond it. Used by the font subsetter. + +using System.Buffers.Binary; + +namespace Filee.Engines.Fonts; + +/// Reads and writes the character-to-glyph map of a font. +internal static class CmapTable +{ + /// Highest code point a format 4 subtable can hold. + private const int BmpLast = 0xFFFF; + + /// + /// The code point → glyph map of the best subtable: full Unicode (3,10 / 0,4 / 0,6) before BMP Unicode (3,1 / 0,x) + /// before symbol (3,0), whose codes are returned as stored (usually U+F020–U+F0FF). Empty without a usable subtable. + /// + public static Dictionary Read(byte[] cmap, out bool symbol) + { + symbol = false; + var r = new BigEndianReader(cmap); + r.Skip(2); // version + int count = r.U16(); + (int Rank, int Offset, bool Symbol) best = (int.MaxValue, -1, false); + for (var i = 0; i < count; i++) + { + int platform = r.U16(); + int encoding = r.U16(); + var offset = r.U32(); + if (offset > cmap.Length - 4) + continue; + var format = BinaryPrimitives.ReadUInt16BigEndian(cmap.AsSpan((int)offset)); + var rank = (platform, encoding) switch + { + (3, 10) or (0, 4) or (0, 6) => 0, + (3, 1) or (0, 3) or (0, 2) or (0, 1) or (0, 0) => 1, + (3, 0) => 2, + _ => int.MaxValue, + }; + if (format is not (4 or 6 or 10 or 12 or 13)) + rank = int.MaxValue; + if (rank < best.Rank) + best = (rank, (int)offset, platform == 3 && encoding == 0); + } + var map = new Dictionary(); + if (best.Offset < 0) + return map; + symbol = best.Symbol; + ReadSubtable(cmap, best.Offset, map); + return map; + } + + private static void ReadSubtable(byte[] cmap, int offset, Dictionary map) + { + var r = new BigEndianReader(cmap, offset); + int format = r.U16(); + switch (format) + { + case 4: + ReadFormat4(cmap, offset, map); + break; + case 6: + r.Skip(4); // length, language + ReadTrimmed(r, r.U16(), r.U16(), map); + break; + case 10: + r.Skip(10); // reserved, length, language + ReadTrimmed(r, r.U32(), r.U32(), map); + break; + case 12 or 13: + r.Skip(10); // reserved, length, language + ReadGroups(r, sameGlyph: format == 13, map); + break; + } + } + + /// Formats 6 and 10: a glyph for each code of one range. + private static void ReadTrimmed(BigEndianReader r, uint first, uint count, Dictionary map) + { + if (count > (uint)r.Remaining / 2) + throw new InvalidDataException("The font's character map is damaged."); + for (uint i = 0; i < count; i++) + Add(map, (int)Math.Min(first + i, int.MaxValue), r.U16()); + } + + /// Formats 12 (consecutive glyphs per group) and 13 (one glyph per group). + private static void ReadGroups(BigEndianReader r, bool sameGlyph, Dictionary map) + { + var groups = r.U32(); + if (groups > (uint)r.Remaining / 12) + throw new InvalidDataException("The font's character map is damaged."); + for (uint g = 0; g < groups; g++) + { + var start = r.U32(); + var end = Math.Min(r.U32(), 0x10FFFF); + var glyph = r.U32(); + for (var c = start; c <= end; c++) + Add(map, (int)c, sameGlyph ? glyph : glyph + (c - start)); + } + } + + private static void ReadFormat4(byte[] cmap, int offset, Dictionary map) + { + var r = new BigEndianReader(cmap, offset + 6); + var segments = r.U16() / 2; + var ends = offset + 14; + var starts = ends + segments * 2 + 2; + var deltas = starts + segments * 2; + var rangeOffsets = deltas + segments * 2; + if (rangeOffsets + segments * 2 > cmap.Length) + throw new InvalidDataException("The font's character map is damaged."); + for (var s = 0; s < segments; s++) + { + int end = BinaryPrimitives.ReadUInt16BigEndian(cmap.AsSpan(ends + s * 2)); + int start = BinaryPrimitives.ReadUInt16BigEndian(cmap.AsSpan(starts + s * 2)); + int delta = BinaryPrimitives.ReadInt16BigEndian(cmap.AsSpan(deltas + s * 2)); + int rangeOffset = BinaryPrimitives.ReadUInt16BigEndian(cmap.AsSpan(rangeOffsets + s * 2)); + for (var c = start; c <= end && c != 0xFFFF; c++) + { + int glyph; + if (rangeOffset == 0) + { + glyph = (c + delta) & 0xFFFF; + } + else + { + var at = rangeOffsets + s * 2 + rangeOffset + (c - start) * 2; + if (at < 0 || at + 2 > cmap.Length) + continue; + glyph = BinaryPrimitives.ReadUInt16BigEndian(cmap.AsSpan(at)); + if (glyph != 0) + glyph = (glyph + delta) & 0xFFFF; + } + Add(map, c, (uint)glyph); + } + } + } + + private static void Add(Dictionary map, int code, uint glyph) + { + if (glyph is > 0 and <= 0xFFFF) + map.TryAdd(code, (int)glyph); + } + + /// + /// A cmap table for : (3,1) format 4 with the BMP (or (3,0) for a symbol font) and, when + /// the map has code points above U+FFFF, (3,10) format 12 with all of them. + /// + public static byte[] Build(IReadOnlyDictionary map, bool symbol = false) + { + var sorted = map.Where(p => p.Key >= 0 && p.Value > 0).OrderBy(p => p.Key).ToList(); + var format4 = Format4([.. sorted.Where(p => p.Key < BmpLast)]); + var full = !symbol && sorted.Any(p => p.Key > BmpLast) ? Format12(sorted) : null; + + var w = new BigEndianWriter(12 + format4.Length + (full?.Length ?? 0)); + w.U16(0); // version + w.U16(full is null ? 1 : 2); + var offset = 4 + 8 * (full is null ? 1 : 2); + w.U16(3); + w.U16(symbol ? 0 : 1); + w.U32((uint)offset); + if (full is not null) + { + w.U16(3); + w.U16(10); + w.U32((uint)(offset + format4.Length)); + } + w.Bytes(format4); + if (full is not null) + w.Bytes(full); + return w.ToArray(); + } + + /// Format 4 with one segment per run of consecutive codes mapped to consecutive glyphs (idDelta only). + private static byte[] Format4(List> map) + { + var segments = new List<(int Start, int End, int Delta)>(); + foreach (var (code, glyph) in map) + { + if (segments.Count > 0 && segments[^1] is var last && last.End == code - 1 && last.Delta == glyph - code) + segments[^1] = (last.Start, code, last.Delta); + else + segments.Add((code, code, glyph - code)); + } + segments.Add((0xFFFF, 0xFFFF, 1)); // the required final segment + + var count = segments.Count; + var length = 16 + 8 * count; + if (length > 0xFFFF) + throw new InvalidDataException("Too many characters for a format 4 character map."); + var selector = count <= 1 ? 0 : System.Numerics.BitOperations.Log2((uint)count); + var searchRange = 2 << selector; + var w = new BigEndianWriter(length); + w.U16(4); + w.U16(length); + w.U16(0); // language + w.U16(count * 2); + w.U16(searchRange); + w.U16(selector); + w.U16(count * 2 - searchRange); + foreach (var s in segments) + w.U16(s.End); + w.U16(0); // reservedPad + foreach (var s in segments) + w.U16(s.Start); + foreach (var s in segments) + w.U16(s.Delta & 0xFFFF); + foreach (var _ in segments) + w.U16(0); // idRangeOffset + return w.ToArray(); + } + + /// Format 12 with one group per run of consecutive codes mapped to consecutive glyphs. + private static byte[] Format12(List> map) + { + var groups = new List<(int Start, int End, int Glyph)>(); + foreach (var (code, glyph) in map) + { + if (groups.Count > 0 && groups[^1] is var last && last.End == code - 1 && last.Glyph + (code - last.Start) == glyph) + groups[^1] = (last.Start, code, last.Glyph); + else + groups.Add((code, code, glyph)); + } + var w = new BigEndianWriter(16 + 12 * groups.Count); + w.U16(12); + w.U16(0); // reserved + w.U32((uint)(16 + 12 * groups.Count)); + w.U32(0); // language + w.U32((uint)groups.Count); + foreach (var g in groups) + { + w.U32((uint)g.Start); + w.U32((uint)g.End); + w.U32((uint)g.Glyph); + } + return w.ToArray(); + } +} diff --git a/src/Filee.Engines/Fonts/FontSubsetter.cs b/src/Filee.Engines/Fonts/FontSubsetter.cs new file mode 100644 index 000000000..c9d853729 --- /dev/null +++ b/src/Filee.Engines/Fonts/FontSubsetter.cs @@ -0,0 +1,240 @@ +// A font subsetter for PDF output: keeps the outlines of the glyphs a document uses and empties the others, without +// renumbering them, so glyph IDs from shaping stay valid against the subset (a "glyph-ID-preserving" subset). The +// glyph count is cut to the highest used glyph + 1, and the tables that only describe dropped glyphs shrink to almost +// nothing once a PDF writer deflates the font. +// +// What the subset keeps: glyf / loca (the used glyphs and the components of composite ones, hinting instructions +// included), head, hhea / hmtx (advances of the used glyphs), maxp, OS/2, a cmap reduced to the used glyphs (PDF +// writers build ToUnicode from it), post without glyph names, name records 0–6, and the hinting tables cvt / fpgm / prep +// / gasp (some CJK fonts build their glyphs with hinting instructions). Layout tables (GSUB, GPOS, GDEF, kern, ...) +// are dropped: the text is already shaped. Vertical metrics and embedded bitmaps are dropped too. +// +// CFF fonts are converted to TrueType outlines first (only the used glyphs, CffToTrueType), so the result is always a +// TrueType font. Variable TrueType fonts lose their variation tables: the subset is the default instance, which is +// what glyf holds. Fonts the subsetter can't handle (CFF2 outlines, colour and bitmap-only fonts) are reported by +// CanSubset. + +using System.Buffers.Binary; + +namespace Filee.Engines.Fonts; + +/// Glyph-ID-preserving TrueType subsetting. +internal static class FontSubsetter +{ + /// Tables copied unchanged into the subset (when the font has them). + private static readonly string[] KeptTables = ["OS/2", "cvt ", "fpgm", "prep", "gasp"]; + + /// Tables whose glyphs are pictures (colour or bitmap): the subset would draw nothing for them. + private static readonly string[] ColorTables = ["COLR", "CBDT", "sbix", "SVG "]; + + /// name IDs kept: copyright, family, subfamily, unique ID, full name, version, PostScript name. + private const int LastKeptNameId = 6; + + /// + /// True when handles the font; otherwise says why (a PDF writer then + /// draws the text another way). + /// + public static bool CanSubset(SfntFont font, out string? reason) + { + if (font["CFF2"] is not null) + reason = "CFF2 (variable PostScript) outlines can't be subset yet."; + else if (ColorTables.Any(font.Tables.ContainsKey)) + reason = "The font draws its glyphs as colour layers or pictures."; + else if (font["glyf"] is null && font["CFF "] is null) + reason = "The font has no glyph outlines."; + else if (font["glyf"] is not null && font["loca"] is null) + reason = "The font has glyph outlines without a 'loca' table."; + else if (font["head"] is null || font["hhea"] is null || font["hmtx"] is null || font["maxp"] is null) + reason = "The font lacks a required table (head, hhea, hmtx or maxp)."; + else + reason = null; + return reason is null; + } + + /// + /// A TrueType font with the outlines of (plus .notdef and the components they use) at + /// their original glyph IDs and every other glyph empty. + /// + /// The font is one refuses. + /// A glyph ID is outside the font. + public static SfntFont Subset(SfntFont font, IEnumerable glyphs, CancellationToken cancellationToken = default) + { + if (!CanSubset(font, out var reason)) + throw new NotSupportedException(reason); + var glyphCount = font.GlyphCount; + var wanted = new HashSet { 0 }; + foreach (var glyph in glyphs) + { + if (glyph < 0 || glyph >= glyphCount) + throw new ArgumentOutOfRangeException(nameof(glyphs), glyph, $"The font has {glyphCount} glyphs."); + wanted.Add(glyph); + } + + var source = font["glyf"] is null ? CffToTrueType.Convert(font, cancellationToken, only: wanted) : font; + cancellationToken.ThrowIfCancellationRequested(); + + var glyf = source.Require("glyf"); + var head = source.Require("head"); + var longLoca = FontTables.IndexToLocFormat(head) != 0; + var loca = GlyfTable.ReadLoca(source.Require("loca"), glyphCount, longLoca, glyf.Length); + var used = Closure(glyf, loca, wanted); + var count = used.Max() + 1; + + var outlines = new byte[count][]; + for (var gid = 0; gid < count; gid++) + outlines[gid] = used.Contains(gid) ? glyf.AsSpan(loca[gid].Offset, loca[gid].Length).ToArray() : []; + var (newGlyf, newLoca, newLongLoca) = GlyfTable.Build(outlines, longLoca); + + var subset = new SfntFont(SfntFont.TrueTypeFlavor); + subset.Tables["glyf"] = newGlyf; + subset.Tables["loca"] = newLoca; + subset.Tables["head"] = FontTables.WithIndexToLocFormat(head, newLongLoca); + subset.Tables["maxp"] = Maxp(source.Require("maxp"), count); + (subset.Tables["hhea"], subset.Tables["hmtx"]) = HorizontalMetrics(source.Require("hhea"), source.Require("hmtx"), used, count); + subset.Tables["post"] = Post(source["post"]); + if (source["cmap"] is { } cmap) + { + var map = CmapTable.Read(cmap, out var symbol); + subset.Tables["cmap"] = CmapTable.Build(map.Where(p => used.Contains(p.Value)).ToDictionary(), symbol); + } + else + { + subset.Tables["cmap"] = CmapTable.Build(new Dictionary()); + } + if (source["name"] is { } name) + subset.Tables["name"] = Name(name); + foreach (var tag in KeptTables) + { + if (source[tag] is { } table) + subset.Tables[tag] = table; + } + return subset; + } + + /// The wanted glyphs plus every glyph their composite glyphs are built from, at any depth. + private static HashSet Closure(byte[] glyf, (int Offset, int Length)[] loca, HashSet wanted) + { + var used = new HashSet(wanted); + var pending = new Stack(wanted); + while (pending.Count > 0) + { + var (offset, length) = loca[pending.Pop()]; + if (length < 10 || BinaryPrimitives.ReadInt16BigEndian(glyf.AsSpan(offset)) >= 0) + continue; + foreach (var component in Components(glyf, offset + 10, offset + length)) + { + if (component < loca.Length && used.Add(component)) + pending.Push(component); + } + } + return used; + } + + /// The glyph IDs of a composite glyph's components (its records start at ). + private static List Components(byte[] glyf, int start, int end) + { + var r = new BigEndianReader(glyf, start, end); + var result = new List(); + ushort flags; + do + { + flags = r.U16(); + result.Add(r.U16()); + r.Skip((flags & TrueTypeGlyph.ArgsAreWords) != 0 ? 4 : 2); + if ((flags & TrueTypeGlyph.HasScale) != 0) + r.Skip(2); + else if ((flags & TrueTypeGlyph.HasXYScale) != 0) + r.Skip(4); + else if ((flags & TrueTypeGlyph.HasTwoByTwo) != 0) + r.Skip(8); + } + while ((flags & TrueTypeGlyph.MoreComponents) != 0); + return result; + } + + private static byte[] Maxp(byte[] maxp, int count) + { + var copy = (byte[])maxp.Clone(); + BinaryPrimitives.WriteUInt16BigEndian(copy.AsSpan(4), (ushort)count); + return copy; + } + + /// + /// hhea and hmtx for glyphs, all with a full metric: the used glyphs keep their advance + /// and side bearing, the others get zeros (which deflate to almost nothing). + /// + private static (byte[] Hhea, byte[] Hmtx) HorizontalMetrics(byte[] hhea, byte[] hmtx, HashSet used, int count) + { + var metrics = FontTables.NumberOfMetrics(hhea); + if (metrics == 0) + throw new InvalidDataException("The font's horizontal metrics are damaged."); + var r = new BigEndianReader(hmtx); + var w = new BigEndianWriter(count * 4); + var advance = 0; + for (var gid = 0; gid < count; gid++) + { + int bearing; + if (gid < metrics) + { + advance = r.U16(); + bearing = r.I16(); + } + else + { + bearing = r.Remaining >= 2 ? r.I16() : 0; + } + var keep = used.Contains(gid); + w.U16(keep ? advance : 0); + w.I16(keep ? bearing : 0); + } + var header = (byte[])hhea.Clone(); + BinaryPrimitives.WriteUInt16BigEndian(header.AsSpan(34), (ushort)count); + return (header, w.ToArray()); + } + + /// post format 3 (no glyph names) with the original header (italic angle, underline, fixed pitch). + private static byte[] Post(byte[]? post) + { + var result = new byte[32]; + post?.AsSpan(0, Math.Min(32, post.Length)).CopyTo(result); + BinaryPrimitives.WriteUInt32BigEndian(result, 0x00030000); + return result; + } + + /// The name table with only the records that identify the font (IDs 0–6), as format 0. + private static byte[] Name(byte[] name) + { + var r = new BigEndianReader(name); + r.Skip(2); // format + int count = r.U16(); + int storage = r.U16(); + var records = new List<(int Platform, int Encoding, int Language, int Id, byte[] Text)>(); + for (var i = 0; i < count; i++) + { + int platform = r.U16(), encoding = r.U16(), language = r.U16(), id = r.U16(), length = r.U16(), offset = r.U16(); + if (id > LastKeptNameId || language >= 0x8000 || storage + offset + length > name.Length) + continue; + records.Add((platform, encoding, language, id, name.AsSpan(storage + offset, length).ToArray())); + } + records.Sort((a, b) => (a.Platform, a.Encoding, a.Language, a.Id).CompareTo((b.Platform, b.Encoding, b.Language, b.Id))); + + var w = new BigEndianWriter(6 + records.Sum(x => 12 + x.Text.Length)); + w.U16(0); + w.U16(records.Count); + w.U16(6 + 12 * records.Count); + var position = 0; + foreach (var record in records) + { + w.U16(record.Platform); + w.U16(record.Encoding); + w.U16(record.Language); + w.U16(record.Id); + w.U16(record.Text.Length); + w.U16(position); + position += record.Text.Length; + } + foreach (var record in records) + w.Bytes(record.Text); + return w.ToArray(); + } +} diff --git a/src/Filee.Engines/Fonts/SfntFont.cs b/src/Filee.Engines/Fonts/SfntFont.cs index fa77cec6e..3d3a98d22 100644 --- a/src/Filee.Engines/Fonts/SfntFont.cs +++ b/src/Filee.Engines/Fonts/SfntFont.cs @@ -127,6 +127,19 @@ public static SfntFont Read(byte[] data, int offset = 0) return font; } + /// Reads a TTF/OTF, or font of a TTC collection. + public static SfntFont ReadFace(byte[] data, int faceIndex) + { + if (faceIndex == 0 || data.Length < 12 || BinaryPrimitives.ReadUInt32BigEndian(data) != CollectionTag) + return faceIndex == 0 ? Read(data) : throw new InvalidDataException("The font file holds only one font."); + var reader = new BigEndianReader(data, 8); + var count = reader.U32(); + if (faceIndex < 0 || faceIndex >= count) + throw new InvalidDataException($"The font collection has {count} fonts."); + reader.Skip(4 * faceIndex); + return Read(data, checked((int)reader.U32())); + } + /// /// Writes the font: directory sorted by tag, table data in the order the OpenType spec recommends, every table /// 4-byte aligned, checksums and head.checkSumAdjustment computed. diff --git a/src/Filee.Engines/Rendering/PageContent.cs b/src/Filee.Engines/Rendering/PageContent.cs new file mode 100644 index 000000000..10529cb05 --- /dev/null +++ b/src/Filee.Engines/Rendering/PageContent.cs @@ -0,0 +1,201 @@ +// The drawing of one page as the layout engine hands it to output: shaped glyph runs with the text they stand for, +// other drawing (paths, pictures) as Skia calls, links and link targets. PdfPageWriter turns pages into a PDF and +// PageContent.Draw into pixels, both from the same record. +// +// Coordinates are PDF points from the page's top-left corner, y downwards (Skia's convention). + +using SkiaSharp; + +namespace Filee.Engines.Rendering; + +/// +/// A shaped run of glyphs in one font and size. gives, per glyph, the UTF-16 index in +/// where the glyph's cluster starts, as HarfBuzz reports it (in visual order, so descending for +/// right-to-left text). Glyph positions are baseline origins in page coordinates. +/// +internal sealed record GlyphRun(PageFont Font, float Size, ushort[] Glyphs, SKPoint[] Positions, string Text, int[] Clusters) +{ + /// Fill colour of the glyphs. + public SKColor Color { get; init; } = SKColors.Black; + + /// Horizontal scale of the glyphs (1 = as designed; 한글's 장평). + public float ScaleX { get; init; } = 1; + + /// Throws when the arrays don't describe the same glyphs or a cluster points outside the text. + public void Validate() + { + if (Positions.Length != Glyphs.Length || Clusters.Length != Glyphs.Length) + throw new ArgumentException("A glyph run needs one position and one cluster per glyph."); + if (Clusters.Any(c => c < 0 || c > Text.Length)) + throw new ArgumentException("A glyph run's cluster points outside its text."); + if (Size <= 0 || !float.IsFinite(Size)) + throw new ArgumentException("A glyph run needs a positive font size."); + } +} + +/// An outline (bookmark) entry: a heading's title and where it is. +/// Nesting level, 0 for top-level entries. +/// Distance of the target from the top of the page, in points. +internal sealed record OutlineEntry(string Title, int PageIndex, float Top, int Level); + +/// Everything drawn on one page, in drawing order. +internal sealed class PageContent(float width, float height) +{ + private readonly List _items = []; + + /// Page width in points. + public float Width { get; } = width; + + /// Page height in points. + public float Height { get; } = height; + + public IReadOnlyList Items => _items; + + /// Draws a glyph run. + public void DrawGlyphs(GlyphRun run) + { + run.Validate(); + _items.Add(new GlyphItem(run)); + } + + /// Draws something other than text (paths, fills, pictures) with Skia. + public void Draw(Action drawing) => _items.Add(new DrawingItem(drawing)); + + /// Makes a link to . + public void LinkToUrl(SKRect area, string url) => _items.Add(new UrlLinkItem(area, url)); + + /// Makes a link to the named destination . + public void LinkToDestination(SKRect area, string name) => _items.Add(new DestinationLinkItem(area, name)); + + /// Defines the named destination at . + public void Destination(SKPoint point, string name) => _items.Add(new DestinationItem(point, name)); + + /// + /// Draws the page onto , glyph runs with the typeface gives for + /// their font (PDF: the subset; pictures: the whole font). + /// + public void Draw(SKCanvas canvas, Func? typeface = null) + { + typeface ??= f => f.Typeface; + foreach (var item in _items) + { + switch (item) + { + case GlyphItem glyphs: + DrawRun(canvas, glyphs.Run, typeface(glyphs.Run.Font)); + break; + case DrawingItem drawing: + canvas.Save(); + drawing.Drawing(canvas); + canvas.Restore(); + break; + case UrlLinkItem link: + canvas.DrawUrlAnnotation(link.Area, link.Url)?.Dispose(); + break; + case DestinationLinkItem link: + canvas.DrawLinkDestinationAnnotation(link.Area, link.Name)?.Dispose(); + break; + case DestinationItem destination: + canvas.DrawNamedDestinationAnnotation(destination.Point, destination.Name)?.Dispose(); + break; + } + } + } + + /// + /// Draws a run as one text blob that carries the run's text and clusters, so Skia's PDF backend writes a ToUnicode + /// map for one-to-one glyphs and /ActualText for the others (ligatures, conjuncts, reordered vowel signs). + /// Right-to-left runs are drawn in logical order (their glyphs reversed, each at its own position), so readers that + /// take text in content order get it the way it was written. + /// + private static void DrawRun(SKCanvas canvas, GlyphRun run, SKTypeface typeface) + { + if (run.Glyphs.Length == 0) + return; + var (glyphs, positions, clusters) = LogicalOrder(run); + var (utf8, offsets) = Utf8Clusters(run.Text, clusters); + + // Unhinted outlines at subpixel positions: what PDF viewers draw, so pictures of a page match its PDF. + using var font = new SKFont(typeface, run.Size) + { + ScaleX = run.ScaleX, + Subpixel = true, + Edging = SKFontEdging.Antialias, + Hinting = SKFontHinting.None, + LinearMetrics = true, + }; + using var builder = new SKTextBlobBuilder(); + var buffer = builder.AllocatePositionedTextRun(font, glyphs.Length, utf8.Length, null); + buffer.SetGlyphs(glyphs); + buffer.SetPositions(positions); + buffer.SetText(utf8); + buffer.SetClusters(offsets); + using var blob = builder.Build(); + if (blob is null) + return; + using var paint = new SKPaint { Color = run.Color, IsAntialias = true }; + canvas.DrawText(blob, 0, 0, paint); + } + + /// + /// The run's arrays in logical order: reversed when the clusters descend (right-to-left). Clusters that are still + /// out of order (a shaper that reorders across clusters) become one cluster spanning the run. + /// + internal static (ushort[] Glyphs, SKPoint[] Positions, int[] Clusters) LogicalOrder(GlyphRun run) + { + var glyphs = run.Glyphs; + var positions = run.Positions; + var clusters = run.Clusters; + if (clusters.Length > 1 && clusters[0] > clusters[^1]) + { + glyphs = [.. glyphs.Reverse()]; + positions = [.. positions.Reverse()]; + clusters = [.. clusters.Reverse()]; + } + for (var i = 1; i < clusters.Length; i++) + { + if (clusters[i] < clusters[i - 1]) + { + clusters = new int[clusters.Length]; + break; + } + } + return (glyphs, positions, clusters); + } + + /// The text as UTF-8 and each glyph's cluster as a byte offset into it (what Skia's text blobs take). + internal static (byte[] Utf8, uint[] Offsets) Utf8Clusters(string text, int[] clusters) + { + var byteOffsets = new int[text.Length + 1]; + var position = 0; + for (var i = 0; i < text.Length; i++) + { + byteOffsets[i] = position; + if (char.IsHighSurrogate(text[i]) && i + 1 < text.Length && char.IsLowSurrogate(text[i + 1])) + { + byteOffsets[++i] = position; + position += 4; + } + else + { + position += text[i] < 0x80 ? 1 : text[i] < 0x800 ? 2 : 3; + } + } + byteOffsets[text.Length] = position; + var utf8 = System.Text.Encoding.UTF8.GetBytes(text); + return (utf8, [.. clusters.Select(c => (uint)byteOffsets[c])]); + } +} + +/// One drawing step of a page. +internal abstract record PageItem; + +internal sealed record GlyphItem(GlyphRun Run) : PageItem; + +internal sealed record DrawingItem(Action Drawing) : PageItem; + +internal sealed record UrlLinkItem(SKRect Area, string Url) : PageItem; + +internal sealed record DestinationLinkItem(SKRect Area, string Name) : PageItem; + +internal sealed record DestinationItem(SKPoint Point, string Name) : PageItem; diff --git a/src/Filee.Engines/Rendering/PageFont.cs b/src/Filee.Engines/Rendering/PageFont.cs new file mode 100644 index 000000000..320bd3591 --- /dev/null +++ b/src/Filee.Engines/Rendering/PageFont.cs @@ -0,0 +1,59 @@ +// A font the page model draws glyphs from: the font file's bytes (the PDF writer subsets them) and a Skia typeface +// over the whole font (for shaping, measuring and drawing PNG pages). + +using Filee.Engines.Fonts; +using SkiaSharp; + +namespace Filee.Engines.Rendering; + +/// One face of a font file, as the layout engine uses it. +internal sealed class PageFont : IDisposable +{ + private SKTypeface? _typeface; + private SfntFont? _sfnt; + + private PageFont(byte[] data, int faceIndex, SKTypeface? typeface) + { + Data = data; + FaceIndex = faceIndex; + _typeface = typeface; + } + + /// The font file (TTF, OTF or TTC). + public byte[] Data { get; } + + /// Index of the face in a TTC collection; 0 otherwise. + public int FaceIndex { get; } + + /// A Skia typeface over the whole font. + public SKTypeface Typeface => _typeface ??= SKTypeface.FromData(SKData.CreateCopy(Data), FaceIndex) + ?? throw new InvalidDataException("Skia can't read the font."); + + /// The font's tables. + public SfntFont Sfnt => _sfnt ??= SfntFont.ReadFace(Data, FaceIndex); + + /// Reads face of a font file. + public static PageFont FromFile(string path, int faceIndex = 0) => FromData(File.ReadAllBytes(path), faceIndex); + + /// A font from the bytes of a font file. + public static PageFont FromData(byte[] data, int faceIndex = 0) + { + if (!SfntFont.IsSfnt(data)) + throw new InvalidDataException("This is not a TrueType or OpenType font."); + return new PageFont(data, faceIndex, null); + } + + /// The font file behind a typeface Skia found (an installed font, a font manager match). + public static PageFont FromTypeface(SKTypeface typeface) + { + using var stream = typeface.OpenStream(out var index) ?? throw new InvalidDataException($"Skia can't read the font data of {typeface.FamilyName}."); + var data = new byte[stream.Length]; + if (stream.Read(data, data.Length) != data.Length) + throw new InvalidDataException($"Skia can't read the font data of {typeface.FamilyName}."); + if (!SfntFont.IsSfnt(data)) + throw new InvalidDataException($"{typeface.FamilyName} is not a TrueType or OpenType font."); + return new PageFont(data, index, null); + } + + public void Dispose() => _typeface?.Dispose(); +} diff --git a/src/Filee.Engines/Rendering/PdfPageWriter.cs b/src/Filee.Engines/Rendering/PdfPageWriter.cs new file mode 100644 index 000000000..1d110e4b4 --- /dev/null +++ b/src/Filee.Engines/Rendering/PdfPageWriter.cs @@ -0,0 +1,186 @@ +// Writes pages of the layout engine's page model as a PDF through Skia's PDF backend (SKDocument.CreatePdf), with +// fonts subset by Filee (FontSubsetter): SkiaSharp's backend has no subsetter of its own and embeds every font file +// whole (13 MB for one line of Malgun Gothic), and draws CFF and variable fonts as Type 3 fonts. +// +// Two passes: the glyphs every page uses are collected per font first, each font is subset once for the whole +// document (glyph IDs unchanged, so the shaped runs draw as they are), and then the pages are drawn with the subset +// typefaces. Skia writes each subset as a Type 0 / CIDFontType2 font with a ToUnicode map from the subset's cmap and +// /ActualText for glyphs that don't map one to one (see PageContent.DrawRun). Links and named destinations are Skia +// annotations; the outline (bookmarks) is added afterwards with PDFsharp, which Skia has no API for. +// +// docs/ENGINES.md ("Filee renderer: PDF output") compares this with a page writer of Filee's own on PDFsharp. + +using Filee.Engines.Fonts; +using PdfSharp.Pdf; +using PdfSharp.Pdf.IO; +using SkiaSharp; + +namespace Filee.Engines.Rendering; + +/// Document information written into the PDF. +internal sealed record PdfDocumentInfo +{ + public string? Title { get; init; } + public string? Author { get; init; } + public string? Subject { get; init; } + + /// Outline (bookmark) entries in document order. + public IReadOnlyList Outline { get; init; } = []; +} + +/// Page model → PDF. +internal static class PdfPageWriter +{ + /// Writes as a PDF to . + /// Pages written, as a fraction of all pages. + public static void Write(Stream output, IReadOnlyList pages, PdfDocumentInfo? info = null, + IProgress? progress = null, CancellationToken cancellationToken = default) + { + info ??= new PdfDocumentInfo(); + using var fonts = SubsetFonts(pages, cancellationToken); + if (info.Outline.Count == 0 && pages.All(p => !NeedsExactSize(p))) + { + WritePages(output, pages, info, fonts, progress, cancellationToken); + return; + } + using var buffer = new MemoryStream(); + WritePages(buffer, pages, info, fonts, progress, cancellationToken); + buffer.Position = 0; + Finish(buffer, output, pages, info.Outline); + } + + /// + /// True when the page size isn't whole points: Skia rounds the media box to whole points (A4 becomes 595 × 842) and + /// draws from its top edge, so the box is set again afterwards. + /// + private static bool NeedsExactSize(PageContent page) => + page.Width != MathF.Round(page.Width) || page.Height != MathF.Round(page.Height); + + private static void WritePages(Stream output, IReadOnlyList pages, PdfDocumentInfo info, SubsetTypefaces fonts, + IProgress? progress, CancellationToken cancellationToken) + { + var metadata = new SKDocumentPdfMetadata + { + Title = info.Title, + Author = info.Author, + Subject = info.Subject, + Creator = "Filee", + Producer = "Filee", + Creation = DateTime.Now, + Modified = DateTime.Now, + RasterDpi = SKDocument.DefaultRasterDpi, + EncodingQuality = 100, + PdfA = false, + }; + using var stream = new SKManagedWStream(output); + using var document = SKDocument.CreatePdf(stream, metadata) ?? throw new InvalidOperationException("Skia can't write PDF."); + for (var i = 0; i < pages.Count; i++) + { + cancellationToken.ThrowIfCancellationRequested(); + var page = pages[i]; + var canvas = document.BeginPage(page.Width, page.Height); + page.Draw(canvas, fonts.For); + document.EndPage(); + progress?.Report((i + 1.0) / pages.Count); + } + document.Close(); + } + + /// One subset typeface per font the pages use. + private static SubsetTypefaces SubsetFonts(IReadOnlyList pages, CancellationToken cancellationToken) + { + var glyphs = new Dictionary>(ReferenceEqualityComparer.Instance); + foreach (var page in pages) + { + foreach (var item in page.Items) + { + if (item is not GlyphItem { Run: var run }) + continue; + if (!glyphs.TryGetValue(run.Font, out var set)) + glyphs[run.Font] = set = []; + foreach (var glyph in run.Glyphs) + set.Add(glyph); + } + } + var result = new SubsetTypefaces(); + foreach (var (font, used) in glyphs) + { + cancellationToken.ThrowIfCancellationRequested(); + result.Add(font, SubsetTypeface(font, used, cancellationToken)); + } + return result; + } + + /// + /// A typeface over the subset of ; the whole font when it can't be subset (Skia then + /// embeds it whole or as a Type 3 font, still with its text). + /// + private static SKTypeface SubsetTypeface(PageFont font, IEnumerable glyphs, CancellationToken cancellationToken = default) + { + if (!FontSubsetter.CanSubset(font.Sfnt, out _)) + return font.Typeface; + var subset = FontSubsetter.Subset(font.Sfnt, glyphs, cancellationToken).Write(); + return SKTypeface.FromData(SKData.CreateCopy(subset)) ?? font.Typeface; + } + + /// + /// What Skia can't write, added to the PDF it wrote: exact page sizes (a media box that ends at the top edge Skia + /// drew from, so nothing moves) and the outline. + /// + private static void Finish(Stream pdf, Stream output, IReadOnlyList pages, IReadOnlyList entries) + { + using var document = PdfReader.Open(pdf, PdfDocumentOpenMode.Modify); + for (var i = 0; i < pages.Count; i++) + { + if (!NeedsExactSize(pages[i])) + continue; + var top = MathF.Round(pages[i].Height); + var box = new PdfArray(document); + foreach (var value in new[] { 0, top - pages[i].Height, pages[i].Width, top }) + box.Elements.Add(new PdfReal(Math.Round(value, 3))); + document.Pages[i].Elements["/MediaBox"] = box; + } + AddOutline(document, pages, entries); + document.Save(output, closeStream: false); + } + + /// Adds outline entries pointing into the pages (y measured from the top edge Skia drew from). + private static void AddOutline(PdfDocument document, IReadOnlyList pages, IReadOnlyList entries) + { + if (entries.Count == 0) + return; + var parents = new List { document.Outlines }; + foreach (var entry in entries) + { + if (entry.PageIndex < 0 || entry.PageIndex >= document.PageCount) + throw new ArgumentException($"The outline entry '{entry.Title}' points to page {entry.PageIndex + 1}, which doesn't exist."); + var level = Math.Clamp(entry.Level, 0, parents.Count - 1); + var outline = parents[level].Add(entry.Title, document.Pages[entry.PageIndex], opened: level == 0); + outline.PageDestinationType = PdfPageDestinationType.Xyz; + outline.Top = MathF.Round(pages[entry.PageIndex].Height) - entry.Top; + outline.Left = 0; + parents.RemoveRange(level + 1, parents.Count - level - 1); + parents.Add(outline.Outlines); + } + document.PageMode = PdfPageMode.UseOutlines; + } + + /// Subset typefaces by font; disposes the ones it made. + private sealed class SubsetTypefaces : IDisposable + { + private readonly Dictionary _typefaces = new(ReferenceEqualityComparer.Instance); + + public void Add(PageFont font, SKTypeface typeface) => _typefaces[font] = typeface; + + public SKTypeface For(PageFont font) => _typefaces.GetValueOrDefault(font) ?? font.Typeface; + + public void Dispose() + { + foreach (var (font, typeface) in _typefaces) + { + if (!ReferenceEquals(typeface, font.Typeface)) + typeface.Dispose(); + } + } + } +} diff --git a/tests/Filee.Engines.Tests/FontSubsetterTests.cs b/tests/Filee.Engines.Tests/FontSubsetterTests.cs new file mode 100644 index 000000000..a50921606 --- /dev/null +++ b/tests/Filee.Engines.Tests/FontSubsetterTests.cs @@ -0,0 +1,245 @@ +// The glyph-ID-preserving font subsetter (Fonts/FontSubsetter.cs) that the Filee renderer's PDF output embeds fonts +// with, and the character map code it uses (Fonts/CmapTable.cs). Inputs are the small OFL test fonts in Fonts/ (see +// FontTests) and fonts changed in code. + +using Filee.Engines.Fonts; +using Filee.Engines.Fonts.Cff; +using SkiaSharp; + +namespace Filee.Engines.Tests; + +public class FontSubsetterTests +{ + private const string TrueType = "FileeTest-TrueType.ttf"; + private const string CidKeyed = "FileeTest-CID.otf"; + private const string NameKeyed = "FileeTest-NameKeyed.otf"; + + private static SfntFont Load(string name) => FontFile.Load(File.ReadAllBytes(Path.Combine(AppContext.BaseDirectory, "Fonts", name))); + + private static SKTypeface Typeface(SfntFont font) => SKTypeface.FromData(SKData.CreateCopy(font.Write()))!; + + private static ushort[] Glyphs(SKTypeface typeface, string text) + { + using var font = new SKFont(typeface, 100); + return font.GetGlyphs(text); + } + + [Fact] + public void TrueType_subset_keeps_used_glyphs_at_their_ids_and_empties_the_rest() + { + var source = Load(TrueType); + using var original = Typeface(source); + var used = Glyphs(original, "Hi").Select(g => (int)g).ToArray(); + + var subset = FontSubsetter.Subset(source, used, TestContext.Current.CancellationToken); + using var typeface = Typeface(subset); + + Assert.Equal(SfntFont.TrueTypeFlavor, subset.Flavor); + Assert.Equal(used.Max() + 1, subset.GlyphCount); + var glyf = subset.Require("glyf"); + var loca = GlyfTable.ReadLoca(subset.Require("loca"), subset.GlyphCount, FontTables.IndexToLocFormat(subset.Require("head")) != 0, glyf.Length); + var sourceGlyf = source.Require("glyf"); + var sourceLoca = GlyfTable.ReadLoca(source.Require("loca"), source.GlyphCount, FontTables.IndexToLocFormat(source.Require("head")) != 0, sourceGlyf.Length); + for (var gid = 0; gid < subset.GlyphCount; gid++) + { + if (gid == 0 || used.Contains(gid)) + { + // The same bytes, padded to four. + var expected = sourceGlyf.AsSpan(sourceLoca[gid].Offset, sourceLoca[gid].Length); + var actual = glyf.AsSpan(loca[gid].Offset, loca[gid].Length); + Assert.True(actual.StartsWith(expected) && !actual[expected.Length..].ContainsAnyExcept((byte)0), $"glyph {gid}"); + } + else + Assert.Equal(0, loca[gid].Length); + } + + // Skia draws the same outlines and advances from the subset, and its cmap knows only the kept characters. + using var a = new SKFont(original, 100); + using var b = new SKFont(typeface, 100); + foreach (var gid in used) + { + using var expected = a.GetGlyphPath((ushort)gid); + using var actual = b.GetGlyphPath((ushort)gid); + Assert.Equal(expected.ToSvgPathData(), actual.ToSvgPathData()); + Assert.Equal(a.GetGlyphWidths([(ushort)gid])[0], b.GetGlyphWidths([(ushort)gid])[0]); + } + Assert.Equal(used.Select(g => (ushort)g), Glyphs(typeface, "Hi")); + Assert.Equal(0, Glyphs(typeface, "F")[0]); + } + + [Fact] + public void Composite_glyphs_bring_their_components() + { + var source = Load(TrueType); + using var original = Typeface(source); + var aacute = Glyphs(original, "Á")[0]; + var components = ComponentsOf(source, aacute); + Assert.NotEmpty(components); + + var subset = FontSubsetter.Subset(source, [(int)aacute], TestContext.Current.CancellationToken); + using var typeface = Typeface(subset); + + var glyf = subset.Require("glyf"); + var loca = GlyfTable.ReadLoca(subset.Require("loca"), subset.GlyphCount, FontTables.IndexToLocFormat(subset.Require("head")) != 0, glyf.Length); + Assert.All(components, c => Assert.True(loca[c].Length > 0, $"component {c} kept")); + using var a = new SKFont(original, 100); + using var b = new SKFont(typeface, 100); + using var expected = a.GetGlyphPath(aacute); + using var actual = b.GetGlyphPath(aacute); + Assert.Equal(expected.ToSvgPathData(), actual.ToSvgPathData()); + } + + [Fact] + public void Subset_keeps_only_what_a_pdf_needs() + { + var source = Load(TrueType); + source.Tables["fvar"] = [0, 1, 0, 0]; + source.Tables["gvar"] = [0, 1, 0, 0]; + source.Tables["DSIG"] = [0, 0, 0, 1]; + + var subset = FontSubsetter.Subset(source, [1], TestContext.Current.CancellationToken); + + Assert.Equal( + new[] { "OS/2", "cmap", "fpgm", "gasp", "glyf", "head", "hhea", "hmtx", "loca", "maxp", "name", "post", "prep", "cvt " }.Where(t => source[t] is not null).Order(StringComparer.Ordinal), + subset.Tables.Keys.Order(StringComparer.Ordinal)); + Assert.Equal(0x00030000u, System.Buffers.Binary.BinaryPrimitives.ReadUInt32BigEndian(subset.Require("post"))); + // The names that identify the font are kept. + using var typeface = Typeface(subset); + Assert.Equal("Filee Test TrueType", typeface.FamilyName); + Assert.True(subset.Require("name").Length < source.Require("name").Length); + } + + [Theory] + [InlineData(CidKeyed, "한글 Filee")] + [InlineData(NameKeyed, "Aae")] + public void Cff_font_becomes_a_truetype_subset_with_the_same_shapes(string sample, string text) + { + var source = Load(sample); + using var original = Typeface(source); + var used = Glyphs(original, text).Distinct().ToArray(); + + var subset = FontSubsetter.Subset(source, used.Select(g => (int)g), TestContext.Current.CancellationToken); + using var typeface = Typeface(subset); + + Assert.Equal(SfntFont.TrueTypeFlavor, subset.Flavor); + Assert.Null(subset["CFF "]); + Assert.Equal(used.Max() + 1, subset.GlyphCount); + using var a = new SKFont(original, 1000); + using var b = new SKFont(typeface, 1000); + foreach (var gid in used.Where(g => g != 0)) + { + using var expected = a.GetGlyphPath(gid); + using var actual = b.GetGlyphPath(gid); + // Quadratic approximations of the cubic curves: the same box within a font unit or two. + var (e, r) = (expected.Bounds, actual.Bounds); + Assert.True(Math.Abs(e.Left - r.Left) < 3 && Math.Abs(e.Top - r.Top) < 3 && Math.Abs(e.Right - r.Right) < 3 && Math.Abs(e.Bottom - r.Bottom) < 3, + $"glyph {gid}: {e} vs {r}"); + Assert.Equal(a.GetGlyphWidths([gid])[0], b.GetGlyphWidths([gid])[0]); + } + } + + [Fact] + public void Fonts_that_cant_be_subset_say_why() + { + var cff2 = Load(CidKeyed); + cff2.Tables["CFF2"] = cff2.Require("CFF "); + cff2.Tables.Remove("CFF "); + var color = Load(TrueType); + color.Tables["COLR"] = new byte[14]; + + Assert.False(FontSubsetter.CanSubset(cff2, out var reason)); + Assert.Contains("CFF2", reason, StringComparison.Ordinal); + Assert.False(FontSubsetter.CanSubset(color, out _)); + Assert.Throws(() => FontSubsetter.Subset(cff2, [1], TestContext.Current.CancellationToken)); + Assert.True(FontSubsetter.CanSubset(Load(TrueType), out _)); + } + + [Fact] + public void Glyph_ids_outside_the_font_are_refused() + { + var font = Load(TrueType); + + Assert.Throws(() => FontSubsetter.Subset(font, [font.GlyphCount], TestContext.Current.CancellationToken)); + Assert.Throws(() => FontSubsetter.Subset(font, [-1], TestContext.Current.CancellationToken)); + } + + [Fact] + public void Cmap_round_trips_bmp_and_supplementary_characters() + { + var map = new Dictionary { ['A'] = 3, ['B'] = 4, ['C'] = 5, ['가'] = 900, ['힣'] = 901, [0x1F600] = 77, [0x20000] = 78, [0x20001] = 79 }; + + var table = CmapTable.Build(map); + var back = CmapTable.Read(table, out var symbol); + + Assert.False(symbol); + Assert.Equal(map.OrderBy(p => p.Key), back.OrderBy(p => p.Key)); + // BMP-only maps need no format 12 subtable. + Assert.Single(CmapTable.Read(CmapTable.Build(new Dictionary { ['A'] = 3 }), out _)); + Assert.Equal(1, System.Buffers.Binary.BinaryPrimitives.ReadUInt16BigEndian(CmapTable.Build(new Dictionary { ['A'] = 3 }).AsSpan(2))); + } + + [Fact] + public void Symbol_cmap_stays_a_symbol_cmap() + { + var table = CmapTable.Build(new Dictionary { [0xF041] = 2 }, symbol: true); + + Assert.Equal(new Dictionary { [0xF041] = 2 }, CmapTable.Read(table, out var symbol)); + Assert.True(symbol); + } + + [Fact] + public void Cmap_of_a_real_font_is_read_like_skia_reads_it() + { + var source = Load(TrueType); + using var typeface = Typeface(source); + using var font = new SKFont(typeface, 10); + + var map = CmapTable.Read(source.Require("cmap"), out _); + + Assert.NotEmpty(map); + foreach (var (code, glyph) in map) + Assert.Equal(glyph, font.GetGlyphs(char.ConvertFromUtf32(code))[0]); + } + + [Fact] + public void Cid_font_matrices_with_their_own_units_per_em_combine_like_rasterizers_do() + { + // Top DICT 1/2048 and a font dictionary 1/1000 (Pretendard and other fonts built that way): 1/2048 overall. + Assert.Equal([1.0 / 2048, 0, 0, 1.0 / 2048, 0, 0], CffFont.Combine([0.001, 0, 0, 0.001, 0, 0], [1.0 / 2048, 0, 0, 1.0 / 2048, 0, 0]), new DoubleComparer()); + // Top DICT 1/1000 and an identity font dictionary (the usual CID layout). + Assert.Equal([0.001, 0, 0, 0.001, 0, 0], CffFont.Combine([1, 0, 0, 1, 0, 0], [0.001, 0, 0, 0.001, 0, 0]), new DoubleComparer()); + // A slanted font dictionary keeps its slant. + Assert.Equal([0.001, 0, 0.0002, 0.001, 0, 0], CffFont.Combine([0.001, 0, 0.0002, 0.001, 0, 0], [0.001, 0, 0, 0.001, 0, 0]), new DoubleComparer()); + } + + /// The components of a composite glyph (direct ones). + private static List ComponentsOf(SfntFont font, int gid) + { + var glyph = GlyfTable.ReadAll(font)[gid]!; + Assert.True(glyph.IsComposite); + var r = new BigEndianReader(glyph.Components); + var result = new List(); + ushort flags; + do + { + flags = r.U16(); + result.Add(r.U16()); + r.Skip((flags & TrueTypeGlyph.ArgsAreWords) != 0 ? 4 : 2); + if ((flags & TrueTypeGlyph.HasScale) != 0) + r.Skip(2); + else if ((flags & TrueTypeGlyph.HasXYScale) != 0) + r.Skip(4); + else if ((flags & TrueTypeGlyph.HasTwoByTwo) != 0) + r.Skip(8); + } + while ((flags & TrueTypeGlyph.MoreComponents) != 0); + return result; + } + + private sealed class DoubleComparer : IEqualityComparer + { + public bool Equals(double x, double y) => Math.Abs(x - y) < 1e-12; + + public int GetHashCode(double obj) => 0; + } +} diff --git a/tests/Filee.Engines.Tests/PdfPageWriterTests.cs b/tests/Filee.Engines.Tests/PdfPageWriterTests.cs new file mode 100644 index 000000000..7bc23ffde --- /dev/null +++ b/tests/Filee.Engines.Tests/PdfPageWriterTests.cs @@ -0,0 +1,506 @@ +// The Filee renderer's PDF output (Rendering/PdfPageWriter.cs): Skia's PDF backend with fonts subset by Filee. The +// acceptance page of the "PDF backend and font subsetting" spike is built here in code: one A4 page with Korean, Latin, +// Arabic and Devanagari paragraphs shaped by HarfBuzz, a link, outline entries and some non-text drawing. It needs +// installed fonts (Malgun Gothic, Segoe UI, Nirmala UI on Windows) and is skipped where they are missing, like the other +// font-dependent tests. Set FILEE_RENDER_OUT to a folder to keep the PDFs for a look in other viewers. + +using System.Runtime.InteropServices; +using System.Text; +using Filee.Engines.Rendering; +using HarfBuzzSharp; +using PdfSharp.Pdf.IO; +using PDFtoImage; +using SkiaSharp; +using UglyToad.PdfPig.Content; +using Buffer = HarfBuzzSharp.Buffer; +using PdfPigDocument = UglyToad.PdfPig.PdfDocument; + +#pragma warning disable CA1416 + +namespace Filee.Engines.Tests; + +public sealed class PdfPageWriterTests : IDisposable +{ + private const float A4Width = 595.28f, A4Height = 841.89f, Margin = 50; + + private const string Korean = + "Filee는 문서를 PDF로 바꿀 때 글꼴 파일 전체를 넣지 않고 실제로 쓴 글자만 골라 담습니다. 맑은 고딕 한 줄 때문에 " + + "십삼 메가바이트짜리 파일이 생기는 일을 막고, 글자는 그대로 검색하고 복사할 수 있게 남깁니다. 한글 문서의 문단과 표, " + + "머리말과 꼬리말이 모두 같은 방식으로 그려지며 화면에서 보는 모양과 인쇄한 모양이 같아야 합니다. " + + "다람쥐 헌 쳇바퀴에 타고파. 키스의 고유조건은 입술끼리 만나야 하고 특별한 기술은 필요치 않다. " + + "닭 잡아서 치킨파티 함. 콩고물과 우유가 들어간 빙수는 차갑고 달콤하다."; + + private const string Latin = + "Filee writes searchable PDF files: every font is embedded as a subset that holds only the glyphs the page uses, " + + "so a page of text stays small and its words can be found, selected and copied. The quick brown fox jumps over " + + "the lazy dog. Sphinx of black quartz, judge my vow — 0123456789 (€, ©, “quotes”, office, " + + "finally, waffle)."; + + private const string Arabic = + "يكتب فيلي ملفات قابلة للبحث، وكل خط فيها مضمّن كمجموعة جزئية تحتوي فقط على الحروف المستخدمة في الصفحة. " + + "هذا نص عربي للتجربة مع التشكيل: بِسْمِ اللَّهِ الرَّحْمَٰنِ الرَّحِيمِ."; + + private const string Devanagari = + "फ़ाइली खोजने योग्य पीडीएफ़ फ़ाइलें लिखता है: हर फ़ॉन्ट का केवल वह हिस्सा जोड़ा जाता है जिसके अक्षर पृष्ठ पर " + + "इस्तेमाल हुए हैं। हिन्दी में संयुक्ताक्षर जैसे क्ष, त्र, ज्ञ और श्र भी सही दिखने चाहिए।"; + + private readonly Shaper _shaper = new(); + + public void Dispose() => _shaper.Dispose(); + + // ───────────────────────── The acceptance page ───────────────────────── + + public static TheoryData KoreanFonts() => new() + { + "malgun.ttf", // TrueType, 13 MB + "NotoSansCJKkr-Regular.otf", // CID-keyed CFF (Noto Sans CJK) + "NotoSansCJK-Regular.ttc", + "NotoSansKR-VF.ttf", // variable TrueType (Windows 11 ships it) + }; + + [Theory] + [MemberData(nameof(KoreanFonts))] + public void Mixed_script_a4_page_is_small_searchable_and_looks_right(string koreanFont) + { + var korean = InstalledFont(koreanFont); + var latin = InstalledFont("segoeui.ttf"); + var devanagari = InstalledFont("Nirmala.ttc", "Nirmala.ttf"); + Assert.SkipUnless(korean is not null && latin is not null && devanagari is not null, + $"needs {koreanFont}, Segoe UI and Nirmala UI installed"); + using var koreanFace = PageFont.FromFile(korean); + using var latinFace = PageFont.FromFile(latin); + using var devanagariFace = PageFont.FromFile(devanagari); + var (page, text, outline) = MixedPage(koreanFace, latinFace, latinFace, devanagariFace); + + var pdf = Write([page], new PdfDocumentInfo { Title = "Mixed scripts", Outline = outline }); + Keep(pdf, $"mixed-{Path.GetFileNameWithoutExtension(koreanFont)}.pdf"); + + // Under 200 KB with every font embedded and subset. + Assert.True(pdf.Length < 200_000, $"{pdf.Length:N0} bytes"); + var fonts = Fonts(pdf); + Assert.Equal(3, fonts.Count); + foreach (var font in fonts) + { + Assert.Equal("/Type0", font.Subtype); + Assert.Equal("/FontFile2", font.File); + Assert.Matches("^/[A-Z]{6}\\+", font.BaseFont); + } + Assert.True(fonts.Sum(f => f.Length) < new FileInfo(korean!).Length / 10); + + // The exact source text, Arabic in logical order, clusters whole. + Assert.Equal(text, ReadBack(pdf)); + + // PDFium draws what Skia draws (rendered at 3× and averaged down, so rasterizer anti-aliasing differences fade). + const int Dpi = 100; + using var pdfium = Conversion.ToImage(pdf, 0, null, new RenderOptions(Dpi: 3 * Dpi, WithAnnotations: false, BackgroundColor: SKColors.White)); + using var skia = Raster(page, 3 * Dpi); + var (ssim, off) = Compare(Downsample(pdfium, 3), Downsample(skia, 3)); + Assert.True(ssim > 0.9 && off < 0.01, $"SSIM {ssim:F4}, {off:P2} of the pixels differ"); + } + + [Fact] + public void Page_size_links_and_outline_are_exact() + { + var latin = InstalledFont("segoeui.ttf", "arial.ttf", "DejaVuSans.ttf", "Arial.ttf"); + Assert.SkipUnless(latin is not null, "no Latin font installed"); + using var face = PageFont.FromFile(latin); + var page = new PageContent(A4Width, A4Height); + page.DrawGlyphs(_shaper.Shape(face, 20, "Chapter one", Margin, 100).Run); + page.LinkToUrl(new SKRect(Margin, 80, 200, 105), "https://github.com/KnifeLemon/Filee"); + page.Destination(new SKPoint(0, 90), "chapter-1"); + var second = new PageContent(400, 300); + second.DrawGlyphs(_shaper.Shape(face, 12, "Back to chapter one", 20, 40).Run); + second.LinkToDestination(new SKRect(20, 28, 160, 44), "chapter-1"); + OutlineEntry[] outline = [new("Chapter one", 0, 80, 0), new("Section", 0, 300, 1), new("Second page", 1, 20, 0)]; + + var pdf = Write([page, second], new PdfDocumentInfo { Title = "Links", Outline = outline }); + + using (var document = PdfPigDocument.Open(pdf)) + { + var first = document.GetPage(1); + Assert.Equal(A4Width, first.Width, 0.01); + Assert.Equal(A4Height, first.Height, 0.01); + Assert.Equal(400, document.GetPage(2).Width, 0.01); + // The text stays where it was drawn, measured from the top edge. + var letter = first.Letters[0]; + Assert.Equal(100, first.Height - letter.StartBaseLine.Y, 0.01); // PdfPig measures from the media box's corner + Assert.Equal("Chapter one", string.Concat(first.Letters.Select(l => l.Value))); + } + + using var pdfSharp = PdfReader.Open(new MemoryStream(pdf), PdfDocumentOpenMode.Import); + var annotations = pdfSharp.Pages[0].Elements.GetArray("/Annots")!; + Assert.Contains(Enumerable.Range(0, annotations.Elements.Count), i => + annotations.Elements.GetDictionary(i)?.Elements.GetDictionary("/A")?.Elements.GetString("/URI") == "https://github.com/KnifeLemon/Filee"); + Assert.NotNull(pdfSharp.Pages[1].Elements.GetArray("/Annots")); + var titles = pdfSharp.Outlines.Select(o => o.Title).ToList(); + Assert.Equal(["Chapter one", "Second page"], titles); + Assert.Equal("Section", Assert.Single(pdfSharp.Outlines[0].Outlines).Title); + } + + [Fact] + public void Cff_font_is_embedded_as_a_truetype_subset() + { + // FileeTest-CID.otf: a CID-keyed CFF subset of Noto Sans KR (see FontTests), so this runs everywhere. + using var face = PageFont.FromFile(Path.Combine(AppContext.BaseDirectory, "Fonts", "FileeTest-CID.otf")); + var page = new PageContent(300, 100); + var (run, _) = _shaper.Shape(face, 24, "Filee 한글 변환", 10, 50); + page.DrawGlyphs(run); + + var pdf = Write([page]); + + var font = Assert.Single(Fonts(pdf)); + Assert.Equal("/Type0", font.Subtype); + Assert.Equal("/FontFile2", font.File); + Assert.Equal("Filee 한글 변환", ReadBack(pdf)); + } + + [Fact] + public void Right_to_left_runs_are_written_in_logical_order() + { + using var face = PageFont.FromFile(Path.Combine(AppContext.BaseDirectory, "Fonts", "FileeTest-TrueType.ttf")); + // A run as a right-to-left shaper reports it: visual order, clusters descending. + var run = new GlyphRun(face, 10, [3, 2, 1], [new(30, 10), new(20, 10), new(10, 10)], "abc", [2, 1, 0]); + + var (glyphs, positions, clusters) = PageContent.LogicalOrder(run); + + Assert.Equal([1, 2, 3], glyphs); + Assert.Equal([new SKPoint(10, 10), new SKPoint(20, 10), new SKPoint(30, 10)], positions); + Assert.Equal([0, 1, 2], clusters); + // Clusters out of order either way become one cluster for the whole run. + Assert.Equal([0, 0, 0], PageContent.LogicalOrder(run with { Clusters = [1, 2, 0] }).Clusters); + } + + [Fact] + public void Clusters_become_utf8_offsets() + { + var (utf8, offsets) = PageContent.Utf8Clusters("aé한😀b", [0, 1, 2, 3, 5]); + + Assert.Equal(Encoding.UTF8.GetBytes("aé한😀b"), utf8); + Assert.Equal([0u, 1u, 3u, 6u, 10u], offsets); + } + + // ───────────────────────── Page building ───────────────────────── + + /// Headings, paragraphs broken at spaces by shaped width, a link and a filled box. + private (PageContent Page, string Text, List Outline) MixedPage(PageFont korean, PageFont latin, PageFont arabic, PageFont devanagari) + { + var page = new PageContent(A4Width, A4Height); + var text = new StringBuilder(); + var outline = new List(); + var y = Margin + 20f; + var accent = new SKColor(0x1F, 0x3A, 0x68); + + void Heading(string title, PageFont font, bool rightToLeft = false) + { + var x = rightToLeft ? A4Width - Margin - _shaper.Width(font, 16, title) : Margin; + page.DrawGlyphs(_shaper.Shape(font, 16, title, x, y, accent).Run); + text.Append(title); + outline.Add(new OutlineEntry(title, 0, y - 16, 0)); + var ruleY = y + 5; + page.Draw(canvas => + { + using var paint = new SKPaint { Color = accent, StrokeWidth = 0.75f, IsAntialias = true }; + canvas.DrawLine(Margin, ruleY, A4Width - Margin, ruleY, paint); + }); + y += 28; + } + + void Paragraph(string body, PageFont font, float size, bool rightToLeft = false) + { + foreach (var line in Lines(font, size, body, A4Width - 2 * Margin)) + { + var x = rightToLeft ? A4Width - Margin - _shaper.Width(font, size, line) : Margin; + page.DrawGlyphs(_shaper.Shape(font, size, line, x, y).Run); + text.Append(line); + y += size * 1.6f; + } + y += 14; + } + + Heading("1. 한국어", korean); + Paragraph(Korean, korean, 11); + Heading("2. English", latin); + Paragraph(Latin, latin, 11); + page.LinkToUrl(new SKRect(Margin, y - 30, Margin + 120, y - 16), "https://github.com/KnifeLemon/Filee"); + Heading("العربية", arabic, rightToLeft: true); + Paragraph(Arabic, arabic, 13, rightToLeft: true); + Heading("4. हिन्दी", devanagari); + Paragraph(Devanagari, devanagari, 12); + var boxTop = y; + page.Draw(canvas => + { + using var fill = new SKPaint { Color = new SKColor(0xE8, 0xEE, 0xF7) }; + canvas.DrawRoundRect(new SKRect(Margin, boxTop, A4Width - Margin, boxTop + 40), 6, 6, fill); + }); + return (page, text.ToString(), outline); + } + + private List Lines(PageFont font, float size, string body, float maxWidth) + { + var lines = new List(); + var current = ""; + foreach (var word in body.Split(' ', StringSplitOptions.RemoveEmptyEntries)) + { + var candidate = current.Length == 0 ? word : current + " " + word; + if (current.Length > 0 && _shaper.Width(font, size, candidate) > maxWidth) + { + lines.Add(current); + current = word; + } + else + { + current = candidate; + } + } + if (current.Length > 0) + lines.Add(current); + return lines; + } + + /// HarfBuzz shaping of one run in one font (the layout engine's shaper is a later step). + private sealed class Shaper : IDisposable + { + private readonly Dictionary _fonts = new(ReferenceEqualityComparer.Instance); + + public float Width(PageFont font, float size, string text) => Shape(font, size, text, 0, 0).Width; + + /// Shapes with the pen starting at x on baseline y (page coordinates). + public (GlyphRun Run, float Width) Shape(PageFont font, float size, string text, float x, float y, SKColor? color = null) + { + var (hb, upem) = For(font); + using var buffer = new Buffer(); + buffer.AddUtf16(text); + buffer.GuessSegmentProperties(); + hb.Shape(buffer); + var infos = buffer.GlyphInfos; + var positions = buffer.GlyphPositions; + var scale = size / upem; + var glyphs = new ushort[infos.Length]; + var points = new SKPoint[infos.Length]; + var clusters = new int[infos.Length]; + var pen = 0f; + for (var i = 0; i < infos.Length; i++) + { + glyphs[i] = (ushort)infos[i].Codepoint; + clusters[i] = (int)infos[i].Cluster; + points[i] = new SKPoint(x + pen + positions[i].XOffset * scale, y - positions[i].YOffset * scale); + pen += positions[i].XAdvance * scale; + } + return (new GlyphRun(font, size, glyphs, points, text, clusters) { Color = color ?? SKColors.Black }, pen); + } + + private (HarfBuzzSharp.Font Font, int Upem) For(PageFont font) + { + if (!_fonts.TryGetValue(font, out var entry)) + { + // A native copy the blob owns: HarfBuzz reads the font data for as long as the face lives. + var memory = Marshal.AllocHGlobal(font.Data.Length); + Marshal.Copy(font.Data, 0, memory, font.Data.Length); + var blob = new Blob(memory, font.Data.Length, MemoryMode.ReadOnly, () => Marshal.FreeHGlobal(memory)); + var face = new Face(blob, font.FaceIndex); + var hb = new HarfBuzzSharp.Font(face); + hb.SetScale(face.UnitsPerEm, face.UnitsPerEm); + hb.SetFunctionsOpenType(); + _fonts[font] = entry = (blob, face, hb); + } + return (entry.Font, entry.Face.UnitsPerEm); + } + + public void Dispose() + { + foreach (var (blob, face, font) in _fonts.Values) + { + font.Dispose(); + face.Dispose(); + blob.Dispose(); + } + } + } + + // ───────────────────────── Checks ───────────────────────── + + private static byte[] Write(IReadOnlyList pages, PdfDocumentInfo? info = null) + { + using var output = new MemoryStream(); + PdfPageWriter.Write(output, pages, info, cancellationToken: TestContext.Current.CancellationToken); + return output.ToArray(); + } + + private static void Keep(byte[] pdf, string name) + { + if (Environment.GetEnvironmentVariable("FILEE_RENDER_OUT") is { Length: > 0 } folder) + { + Directory.CreateDirectory(folder); + File.WriteAllBytes(Path.Combine(folder, name), pdf); + } + } + + /// A font file from the system's or the user's font folders, by file name; null when none is there. + private static string? InstalledFont(params string[] names) + { + var folders = new List + { + Environment.GetFolderPath(Environment.SpecialFolder.Fonts), + Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Microsoft", "Windows", "Fonts"), + "/usr/share/fonts", "/usr/local/share/fonts", "/Library/Fonts", "/System/Library/Fonts", + Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), "Library", "Fonts"), + }; + foreach (var folder in folders.Where(f => f.Length > 0 && Directory.Exists(f))) + { + foreach (var name in names) + { + var match = Directory.EnumerateFiles(folder, name, new EnumerationOptions { RecurseSubdirectories = true, MatchCasing = MatchCasing.CaseInsensitive, IgnoreInaccessible = true }) + .FirstOrDefault(); + if (match is not null) + return match; + } + } + return null; + } + + private sealed record FontInfo(string BaseFont, string Subtype, string? File, int Length); + + /// The fonts of every page: Type 0 fonts are described by their descendant. + private static List Fonts(byte[] pdf) + { + using var document = PdfReader.Open(new MemoryStream(pdf), PdfDocumentOpenMode.Import); + var result = new List(); + foreach (var page in document.Pages) + { + if (page.Resources.Elements.GetDictionary("/Font") is not { } fonts) + continue; + foreach (var key in fonts.Elements.Keys) + { + var font = fonts.Elements.GetDictionary(key)!; + var subtype = font.Elements.GetName("/Subtype"); + var described = subtype == "/Type0" ? font.Elements.GetArray("/DescendantFonts")!.Elements.GetDictionary(0)! : font; + var descriptor = described.Elements.GetDictionary("/FontDescriptor"); + var file = new[] { "/FontFile", "/FontFile2", "/FontFile3" }.FirstOrDefault(k => descriptor?.Elements.GetDictionary(k) is not null); + result.Add(new FontInfo(font.Elements.GetName("/BaseFont"), subtype, file, file is null ? 0 : descriptor!.Elements.GetDictionary(file)!.Stream.Length)); + } + } + return result; + } + + /// The text in content-stream order, the glyphs of each /ActualText span replaced by its text. + private static string ReadBack(byte[] pdf) + { + using var document = PdfPigDocument.Open(pdf); + var result = new StringBuilder(); + foreach (var page in document.GetPages()) + { + var spans = new Dictionary(ReferenceEqualityComparer.Instance); + void Walk(IEnumerable elements) + { + foreach (var element in elements) + { + if (element.ActualText is null) + { + Walk(element.Children); + continue; + } + foreach (var letter in All(element)) + spans.TryAdd(letter, element); + } + } + Walk(page.GetMarkedContents()); + var done = new HashSet(ReferenceEqualityComparer.Instance); + foreach (var letter in page.Letters) + { + if (!spans.TryGetValue(letter, out var span)) + result.Append(letter.Value); + else if (done.Add(span)) + result.Append(span.ActualText); + } + } + return result.ToString(); + + static IEnumerable All(MarkedContentElement element) => element.Letters.Concat(element.Children.SelectMany(All)); + } + + private static SKBitmap Raster(PageContent page, int dpi) + { + var scale = dpi / 72f; + var bitmap = new SKBitmap((int)Math.Round(page.Width * scale), (int)Math.Round(page.Height * scale)); + using var canvas = new SKCanvas(bitmap); + canvas.Clear(SKColors.White); + canvas.Scale(scale); + page.Draw(canvas); + return bitmap; + } + + /// Grey levels averaged over ² blocks. + private static (int Width, int Height, double[] Grey) Downsample(SKBitmap bitmap, int factor) + { + using var grey = new SKBitmap(bitmap.Width, bitmap.Height, SKColorType.Gray8, SKAlphaType.Opaque); + using (var canvas = new SKCanvas(grey)) + canvas.DrawBitmap(bitmap, 0, 0); + var pixels = grey.GetPixelSpan(); + var stride = grey.RowBytes; + int width = bitmap.Width / factor, height = bitmap.Height / factor; + var result = new double[width * height]; + for (var y = 0; y < height; y++) + { + for (var x = 0; x < width; x++) + { + var sum = 0; + for (var j = 0; j < factor; j++) + for (var i = 0; i < factor; i++) + sum += pixels[(y * factor + j) * stride + x * factor + i]; + result[y * width + x] = (double)sum / (factor * factor); + } + } + return (width, height, result); + } + + /// + /// Mean SSIM over 8×8 windows (step 4) that have ink in either picture, and the share of pixels whose grey levels + /// differ by more than 64. + /// + private static (double Ssim, double Off) Compare((int Width, int Height, double[] Grey) a, (int Width, int Height, double[] Grey) b) + { + int w = Math.Min(a.Width, b.Width), h = Math.Min(a.Height, b.Height); + double At((int Width, int Height, double[] Grey) p, int x, int y) => p.Grey[y * p.Width + x]; + const double C1 = 6.5025, C2 = 58.5225; + double sum = 0; + var windows = 0; + for (var y = 0; y + 8 <= h; y += 4) + { + for (var x = 0; x + 8 <= w; x += 4) + { + double ma = 0, mb = 0; + for (var j = 0; j < 8; j++) + for (var i = 0; i < 8; i++) + { + ma += At(a, x + i, y + j); + mb += At(b, x + i, y + j); + } + ma /= 64; + mb /= 64; + double va = 0, vb = 0, cov = 0; + for (var j = 0; j < 8; j++) + for (var i = 0; i < 8; i++) + { + var da = At(a, x + i, y + j) - ma; + var db = At(b, x + i, y + j) - mb; + va += da * da; + vb += db * db; + cov += da * db; + } + if (va + vb < 1e-9 && ma > 250 && mb > 250) + continue; + va /= 63; + vb /= 63; + cov /= 63; + sum += (2 * ma * mb + C1) * (2 * cov + C2) / ((ma * ma + mb * mb + C1) * (va + vb + C2)); + windows++; + } + } + var off = 0; + for (var y = 0; y < h; y++) + for (var x = 0; x < w; x++) + if (Math.Abs(At(a, x, y) - At(b, x, y)) > 64) + off++; + return (windows == 0 ? 1 : sum / windows, (double)off / (w * h)); + } +}