diff --git a/docs/hermes-base-verification.md b/docs/hermes-base-verification.md index 8a4d93b..776a904 100644 --- a/docs/hermes-base-verification.md +++ b/docs/hermes-base-verification.md @@ -37,7 +37,9 @@ v98 的 shape 索引和 offset 一样只用于定位(delta 可能重排 shape 为什么不能按 dump 的整段文本比:Hermes 的缓冲区构建器会**重叠/去重**序列化后的字面量——一个字面量的最后一个值字节可以同时是下一个字面量的 tag 字节(模糊测试实测:`61 52 | cd 09 b3 05 11`,前一段以 `[String 82]` 结尾,后一条指令的 offset 正指向 `52`)。顺序解析整段缓冲区(hermesc 的 dump 就是这么打印的)从这里开始失步,之后的条目全是噪声;delta 构建的 id 宽度不同,重叠位置也不同,于是两段"噪声"在某处不一致就被判为差异。2026-09-10 的 20 轮冒烟模糊测试里 3 次误杀全部源于此,改按指令比较后全部等价。无法读二进制缓冲区(文件结构不识别)时两侧一起比较整段文本,仅辅助诊断;即使文本相等也返回 `dump-failed` 并回退 plain,不能以丢失 offset/count 的文本确认等价。结果里 `literals: 'buffer'` 标明这一点。 -`normalizeDisassemblyLine` 只折叠表示层差异:按指令解析后的字面量地址、已知宽度后缀、引号外的列对齐空白、switch 表的物理偏移(含经典 `SwitchImm`)、debug 偏移。字符串内部的连续空格、跳转目标标签、寄存器均保留。未知 string ID、无法解码的字面量、未知 buffer 操作数形态直接失败;两侧都无法解析也不等价。 +`normalizeDisassemblyLine` 只折叠表示层差异:按指令解析后的字面量地址、已知宽度后缀、引号外的列对齐空白、switch 表的物理偏移(含经典 `SwitchImm`)、debug 偏移、`DefineOwnById*` 的字符串操作数。字符串内部的连续空格、跳转目标标签、寄存器均保留。 + +**`DefineOwnById*` 的字符串操作数为什么只能丢给 raw**:Hermes 的 `BytecodeList.def` 只给 `DefineOwnByIdLong` 标了字符串操作数,短形式没标。于是同一条指令有两种 pretty 打印:普通编译 id 小、走短形式,打印**裸 id**;base 编译继承了 base 的字符串表,id 溢出 16 位后改用 Long 形式,打印**文本**,而 pretty 的文本又按显示预算截断(`equivalenceCheckPropertyName` 打成 `"equivalenceCheckP"...`)。两种表示互相还原不了——这个预算对非 ASCII 还会静默截断且不加 `...`(实测 `ab中` 打成 `"ab"`、`abcdefghijklmnop中` 打成 `"abcdefghi"`),所以还原出的完整名字永远等不上打印出来的名字。归一化因此把该操作数整个折成 ``,属性名交给 raw 核对按二进制字符串表全量比较(`hermes-raw.ts` 的 `STRING_OPERANDS` 正是为此显式补了 `DefineOwnById`)。2026-09-22 之前这里会把任何超过显示预算的属性名判成差异,线上因此误杀过 base。未知 string ID、无法解码的字面量、未知 buffer 操作数形态直接失败;两侧都无法解析也不等价。 **原始操作数核对**:`hermes-raw.ts` 从 raw dump 读取指令起点和操作数类型,并检查操作数与 HBC 字节一致、指令覆盖完整函数体。字符串从 small/overflow string table 与 string storage 按完整 ASCII/UTF-16 code unit 解码;BigInt、正则和 double 读取真实字节(保留 `-0` 和尾部精度);函数引用保留索引,与顺序对齐的函数表共同检查,同名函数不能互换。地址映射为目标指令序号;整数和字符串 switch 从二进制恢复 case 值与目的地。函数运行时 flags 和参数/寄存器等字段也参与比较,剔除的仅是物理地址、debug presence 与 compact/overflow 表示。 diff --git a/docs/hermes-review-follow-up.md b/docs/hermes-review-follow-up.md new file mode 100644 index 0000000..604207b --- /dev/null +++ b/docs/hermes-review-follow-up.md @@ -0,0 +1,52 @@ +# PR #85 review follow-up + +## Changes + +- `CacheNewObject` now resolves its shape index into the ordered, fully decoded + property keys in the pretty pass. Both registers and the per-function cache + index remain part of the comparison. Unknown shapes, undecodable keys and + unsupported operands fail closed. With no binary resolver, the shape is only + folded for diagnostics; the existing binary-data and raw-audit gates still + forbid text-only equivalence. +- Fuzz success requires a positive integer round count, all requested positive + comparisons, zero compilation/comparison failures, at least one effective + planted difference, and no missed or unbuildable negatives. Optimized-away + differences do not count. The last round also attempts a negative, so runs + shorter than ten rounds are not structurally unable to exercise rejection. +- New regressions cover shape-index relocation, key changes, key order, + register/cache preservation, malformed/missing references, text-only fallback, + and fuzz runs with zero or incomplete useful coverage. +- Enforcing that gate exposed a pre-existing generator defect in the HBC 96 + CI run (seed 96, round 41): the quoted-string regex matched a gap after an + escaped quote and inserted `~` before a quoted property name. String mutation + and negative planting now use real string-token boundaries from the already + installed Babel parser and JSON-encode replacements. The minimized regression + also checks comments, regexps, templates, escapes and UTF-16 source offsets. + The failure gate remains strict; the generator is fixed rather than skipping + the failing round or changing the seed. + +## Scope and evidence + +The CacheNewObject regression uses synthetic instruction bytes and binary literal +sections to exercise the production pretty and raw normalizers. It does not claim +that a source-level fixture was compiled into CacheNewObject by a real hermesc. +The existing pinned HBC 96/98 compiler CI remains required before merging. + +Unknown-opcode semantic coverage remains a separate follow-up, not a verified +current-compiler false-acceptance bug. Before accepting additional compiler +snapshots, audit their complete opcode/operand schemas and referenced sections. +Do not remove the whole pretty pass: information such as exception-handler tables +is not currently covered by the raw instruction normalizer alone. + +Local validation in the authoring environment is limited to Node.js execution of +transpiled helpers and synthetic fixtures; Bun and hermesc are not installed. +The PR description records the subsequent CI state separately. + +## CodeRabbit argument-validation follow-up + +Only an omitted `--rounds` flag selects the default of 200. A flag without a +value, an empty value, or a following option is rejected with exit code 2 before +creating the output directory or invoking the compiler. Four CLI regressions +use an executable compiler fixture with a call marker to verify those side +effects do not occur; the valid-count control verifies the marker does work. +Argument-reading, compilation and run-entry functions now document their contracts. diff --git a/scripts/fuzz-hermes-base.ts b/scripts/fuzz-hermes-base.ts index 597d3ad..801e8c1 100644 --- a/scripts/fuzz-hermes-base.ts +++ b/scripts/fuzz-hermes-base.ts @@ -9,9 +9,10 @@ * `normalizeDisassemblyLine` + a unit test) or a real hermesc delta-mode bug * (report upstream; record it in docs/hermes-base-verification.md §3). * - * Every tenth round also plants a one-literal change into the delta build and - * asserts the check still catches it, so a rule that folds too much shows up - * here as well. + * Every tenth round and the final round also plant a one-literal change into + * the delta build and assert the check still catches it, so a rule that folds + * too much shows up here as well. Success requires all requested comparisons, + * zero compilation failures and at least one effective planted difference. * * HERMESC= bun scripts/fuzz-hermes-base.ts [--rounds N] [--seed S] * [--out DIR] [--verbose] @@ -20,7 +21,7 @@ * cases (both sources, all three HBC files) are kept under --out * (default: a fresh temp dir, printed at the end); passing cases are deleted. * Exit code: 0 when every round was equivalent and every planted change was - * caught, 1 otherwise. + * caught with useful coverage, 1 otherwise; invalid arguments exit 2. */ import { spawnSync } from 'node:child_process'; import fs from 'fs-extra'; @@ -28,18 +29,28 @@ import os from 'os'; import path from 'path'; import { compareHermesBytecode } from '../src/utils/hermes-base'; +import { fuzzStringLiterals } from './hermes-fuzz-literals'; +import { hermesFuzzSucceeded } from './hermes-fuzz-result'; // --------------------------------------------------------------------------- // arguments // --------------------------------------------------------------------------- +/** Read a flag's next token; undefined alone does not imply flag omission. */ function argValue(name: string): string | undefined { const index = process.argv.indexOf(`--${name}`); if (index < 0) return undefined; return process.argv[index + 1]; } -const ROUNDS = Number(argValue('rounds') ?? 200); +// Only an omitted flag uses the default; a missing value becomes NaN. +const ROUNDS = Number( + process.argv.includes('--rounds') ? argValue('rounds') : 200, +); +if (!Number.isSafeInteger(ROUNDS) || ROUNDS <= 0) { + console.error('--rounds must be a positive safe integer'); + process.exit(2); +} const SEED = Number(argValue('seed') ?? Date.now() % 2 ** 31); const VERBOSE = process.argv.includes('--verbose'); const OUT_DIR = @@ -471,13 +482,17 @@ class Gen { break; } case 1: { - // change one string literal (a real literal: quote, body without - // an unescaped quote or backslash, same quote — never the gap - // between two literals) - const literals = [...text.matchAll(/(["'])([^"'\\\n]{1,40})\1/g)]; + // A regex can match the gap after an escaped closing quote. Use + // parser offsets and encode the replacement as one complete token. + const literals = fuzzStringLiterals(text).filter( + ({ value }) => value.length > 0 && value.length <= 40, + ); if (literals.length > 0) { - const m = this.rng.pick(literals); - text = `${text.slice(0, m.index)}${m[1]}${m[2]}~${m[1]}${text.slice((m.index ?? 0) + m[0].length)}`; + const literal = this.rng.pick(literals); + text = + text.slice(0, literal.start) + + JSON.stringify(`${literal.value}~`) + + text.slice(literal.end); } break; } @@ -510,14 +525,17 @@ class Gen { /** identical to `source` except one string literal value — must be caught */ /** `marker`: the new string, to tell whether it survived the optimizer */ plantDifference(source: string): { source: string; marker: string } | null { - const literals = [...source.matchAll(/(["'])([A-Za-z]{3,20})\1/g)]; + const literals = fuzzStringLiterals(source).filter(({ value }) => + /^[A-Za-z]{3,20}$/.test(value), + ); if (literals.length === 0) return null; const target = this.rng.pick(literals); - const before = source.slice(0, target.index); - const after = source.slice((target.index ?? 0) + target[0].length); - const marker = `${target[2]}Z`; + const marker = `${target.value}Z`; return { - source: `${before}${target[1]}${marker}${target[1]}${after}`, + source: + source.slice(0, target.start) + + JSON.stringify(marker) + + source.slice(target.end), marker, }; } @@ -527,6 +545,7 @@ class Gen { // compile + compare // --------------------------------------------------------------------------- +/** Compile a generated source, returning diagnostics on compiler failure. */ function compile( input: string, out: string, @@ -558,6 +577,7 @@ interface Finding { count: number; } +/** Run seeded comparisons and exit successfully only with effective coverage. */ async function main() { const rng = new Rng(SEED); const gen = new Gen(rng); @@ -574,6 +594,7 @@ async function main() { let planted = 0; let plantedMissed = 0; let plantedFolded = 0; + let plantedCompileErrors = 0; const started = Date.now(); for (let round = 0; round < ROUNDS; round++) { @@ -634,14 +655,25 @@ async function main() { console.log(`round ${round}: dump failed — ${outcome.detail}`); } - // detection check: a planted one-literal change must be rejected - if (round % 10 === 9) { + // Include the last round so even a short run attempts a negative case. + if (round % 10 === 9 || round === ROUNDS - 1) { const wrong = gen.plantDifference(next); if (wrong) { const wrongJs = path.join(dir, 'wrong.js'); const wrongHbc = path.join(dir, 'wrong.delta.hbc'); fs.writeFileSync(wrongJs, wrong.source); - if (!compile(wrongJs, wrongHbc, [`-base-bytecode=${baseHbc}`])) { + const wrongError = compile(wrongJs, wrongHbc, [ + `-base-bytecode=${baseHbc}`, + ]); + if (wrongError) { + plantedCompileErrors++; + keep = true; + fs.writeFileSync( + path.join(dir, 'planted-compile-error.txt'), + wrongError, + ); + console.log(`round ${round}: planted compile error (kept in ${dir})`); + } else { // The literal may sit in code the optimizer removes or folds // (`!'x'`, an unreachable switch case — Static Hermes folds far more // than classic hermesc). Then both builds are really equivalent and @@ -649,6 +681,7 @@ async function main() { // of the check under test (ASCII strings are stored as is). if (!fs.readFileSync(wrongHbc).includes(wrong.marker)) { plantedFolded++; + keep = true; if (VERBOSE) { console.log( `round ${round}: planted "${wrong.marker}" optimized away`, @@ -674,6 +707,9 @@ async function main() { } } } + } else { + // Keep the input to diagnose a run with no effective negative cases. + keep = true; } } @@ -681,17 +717,22 @@ async function main() { } const seconds = ((Date.now() - started) / 1000).toFixed(1); + const different = [...findings.values()].reduce((n, f) => n + f.count, 0); console.log(''); console.log(`rounds: ${ROUNDS} in ${seconds}s (seed ${SEED})`); console.log(`equivalent: ${equivalent}`); - console.log( - `different: ${[...findings.values()].reduce((n, f) => n + f.count, 0)} (${findings.size} unique)`, - ); + console.log(`different: ${different} (${findings.size} unique)`); console.log(`dump failed: ${dumpFailed}`); console.log(`compile errors (generator): ${compileErrors}`); + console.log(`planted compile errors: ${plantedCompileErrors}`); console.log( `planted differences: ${planted}, missed: ${plantedMissed} (${plantedFolded} more optimized away, not counted)`, ); + if (planted === 0) { + console.error( + 'No effective planted difference was checked; coverage is insufficient.', + ); + } if (findings.size > 0) { console.log(''); console.log('unique differences (first occurrence, reproduction dir):'); @@ -700,7 +741,16 @@ async function main() { console.log(` ${f.detail}`); } } - const ok = findings.size === 0 && dumpFailed === 0 && plantedMissed === 0; + const ok = hermesFuzzSucceeded({ + rounds: ROUNDS, + equivalent, + different, + dumpFailed, + compileErrors, + planted, + plantedMissed, + plantedCompileErrors, + }); if (!ok) console.log(`\nfailing cases kept under ${OUT_DIR}`); else if (!argValue('out')) fs.removeSync(OUT_DIR); process.exit(ok ? 0 : 1); diff --git a/scripts/hermes-fuzz-literals.ts b/scripts/hermes-fuzz-literals.ts new file mode 100644 index 0000000..9ffc0b3 --- /dev/null +++ b/scripts/hermes-fuzz-literals.ts @@ -0,0 +1,28 @@ +import { parse } from '@babel/parser'; + +export interface FuzzStringLiteral { + start: number; + end: number; + value: string; +} + +/** Locate actual JS strings, never the gap between two closing/opening quotes. */ +export function fuzzStringLiterals(source: string): FuzzStringLiteral[] { + const { tokens } = parse(source, { + sourceType: 'script', + tokens: true, + errorRecovery: true, + }); + if (!tokens) throw new Error('Parser did not return string-token data'); + const literals: FuzzStringLiteral[] = []; + for (const token of tokens) { + if ( + typeof token.type === 'object' && + token.type.label === 'string' && + typeof token.value === 'string' + ) { + literals.push({ start: token.start, end: token.end, value: token.value }); + } + } + return literals; +} diff --git a/scripts/hermes-fuzz-result.ts b/scripts/hermes-fuzz-result.ts new file mode 100644 index 0000000..46438d6 --- /dev/null +++ b/scripts/hermes-fuzz-result.ts @@ -0,0 +1,26 @@ +export interface HermesFuzzSummary { + rounds: number; + equivalent: number; + different: number; + dumpFailed: number; + compileErrors: number; + planted: number; + plantedMissed: number; + plantedCompileErrors: number; +} + +/** A green run must contain all requested comparisons and an effective negative. */ +export function hermesFuzzSucceeded(summary: HermesFuzzSummary): boolean { + return ( + Number.isSafeInteger(summary.rounds) && + summary.rounds > 0 && + summary.equivalent === summary.rounds && + summary.different === 0 && + summary.dumpFailed === 0 && + summary.compileErrors === 0 && + Number.isSafeInteger(summary.planted) && + summary.planted > 0 && + summary.plantedMissed === 0 && + summary.plantedCompileErrors === 0 + ); +} diff --git a/src/locales/en.ts b/src/locales/en.ts index 0b2ca3d..34654f2 100644 --- a/src/locales/en.ts +++ b/src/locales/en.ts @@ -4,7 +4,7 @@ export default { accountUnknown: 'Unknown', accountNoExpiry: 'No expiry specified', accountRenewalWarning: - '[Warning] Your Pushy paid plan expires in {{days}} days. Renew soon to avoid service disruption: {{url}}', + '[Notice] Your Pushy paid plan expires in {{days}} days. Renew soon to avoid service disruption: {{url}}', accountLookupFailed: '[Warning] Unable to retrieve account information. Continuing with the command.', addedToGitignore: 'Added {{line}} to .gitignore', diff --git a/src/locales/zh.ts b/src/locales/zh.ts index 02c4f2f..69c2a89 100644 --- a/src/locales/zh.ts +++ b/src/locales/zh.ts @@ -4,7 +4,7 @@ export default { accountUnknown: '未知', accountNoExpiry: '未设置有效期', accountRenewalWarning: - '[警告] 您的 Pushy 付费套餐将在 {{days}} 天内到期,请及时续费,以免影响服务:{{url}}', + '[注意] 您的 Pushy 付费套餐将在 {{days}} 天内到期,请及时续费,以免影响服务:{{url}}', accountLookupFailed: '[警告] 暂时无法获取账号信息,将继续执行命令。', addedToGitignore: '已将 {{line}} 添加到 .gitignore', androidCrunchPngsWarning: diff --git a/src/utils/hermes-base.ts b/src/utils/hermes-base.ts index 93971a8..e6bbe4c 100644 --- a/src/utils/hermes-base.ts +++ b/src/utils/hermes-base.ts @@ -19,6 +19,7 @@ import { PassThrough, Readable } from 'stream'; import { pipeline } from 'stream/promises'; import { tempDir } from './constants'; import { getHbcVersion } from './hbcTransform'; +import { normalizeCachedObjectInstruction } from './hermes-cached-object'; import { type LiteralBuffers, LiteralResolver, @@ -1081,6 +1082,9 @@ export function normalizeDisassemblyLine( if (opcode === 'Offset' && line.startsWith('Offset in debug table', indent)) { return null; } + if (opcode === 'CacheNewObject') { + return normalizeCachedObjectInstruction(line, literals); + } let m: RegExpExecArray | null; if (opcode.startsWith('New') && opcode.includes('WithBuffer')) { // v98's AndParent form takes the parent object in a second register, @@ -1112,11 +1116,26 @@ export function normalizeDisassemblyLine( if (m) return `${m[1]} ${m[3]}${normalizeOperandSpacing(m[4])}`; } if (opcode.startsWith('DefineOwnById')) { - m = /^(\s*DefineOwnById\w*\s+r\d+, r\d+, \d+, )(\d+)$/.exec(line); + // BytecodeList.def annotates the string operand of DefineOwnByIdLong but + // not of DefineOwnById, so one instruction has two pretty renderings: the + // plain compile keeps small ids and prints the bare id, while a base + // compile inherits the base's string table, spills past 16 bits, picks the + // Long form and prints the *text* — cut to hermesc's display budget + // (`"equivalenceCheckP"...` for `equivalenceCheckPropertyName`). Neither side can + // be turned into the other: that budget also drops non-ASCII silently and + // without a marker (`ab\u4e2d` prints as `"ab"`), so a resolved name never + // equals a printed one. The operand is folded away here and the property + // name is compared by the raw audit, which decodes it from the string + // table in full (STRING_OPERANDS supplies the missing annotation). + m = /^(\s*DefineOwnById\w*\s+r\d+, r\d+, \d+, )(\d+)?.*$/.exec(line); if (m) { - const text = strings.get(Number(m[2])); - if (text === undefined) throw new Error(`unresolved string id ${m[2]}`); - line = `${m[1]}${JSON.stringify(text)}`; + // An id form whose id is not in the table means the string table was + // not read at all; that still fails closed rather than folding an + // operand nothing could resolve. + if (m[2] !== undefined && !strings.has(Number(m[2]))) { + throw new Error(`unresolved string id ${m[2]}`); + } + line = `${m[1]}`; } } // Operand-width variants of one instruction (GetByIdShort/GetById/GetByIdLong, diff --git a/src/utils/hermes-cached-object.ts b/src/utils/hermes-cached-object.ts new file mode 100644 index 0000000..1dcffbf --- /dev/null +++ b/src/utils/hermes-cached-object.ts @@ -0,0 +1,22 @@ +import type { LiteralResolver } from './hermes-literals'; + +/** Normalize the shape reference, not the registers or the per-function cache. */ +export function normalizeCachedObjectInstruction( + line: string, + literals?: LiteralResolver, +): string { + const m = + /^(\s*)CacheNewObject\s+(r\d+),\s*(r\d+),\s*(\d+),\s*(\d+)\s*$/.exec(line); + if (!m) throw new Error(`unsupported cached object operands: ${line.trim()}`); + const prefix = `${m[1]}CacheNewObject ${m[2]}, ${m[3]}, `; + // Diagnostic-only fallback: compareHermesBytecode still requires readable + // binary data and a successful raw audit before returning equivalent. + if (!literals) return `${prefix}, ${m[5]}`; + const shape = literals.shape(Number(m[4])); + const keys = shape && literals.objectKeys(shape.keyOffset, shape.count); + if (!shape || !keys) { + throw new Error(`undecodable cached object shape ${m[4]}`); + } + // JSON preserves key order, complete string contents and significant spaces. + return `${prefix}keys=${JSON.stringify(keys)}, ${m[5]}`; +} diff --git a/tests/hermes-base.test.ts b/tests/hermes-base.test.ts index 78a49ad..eff00d1 100644 --- a/tests/hermes-base.test.ts +++ b/tests/hermes-base.test.ts @@ -662,9 +662,22 @@ describe('helpers', () => { expect( normalizeDisassemblyLine(' JStrictEqualLong L12, r1, r2', strings), ).toBe(' JStrictEqual L12, r1, r2'); + // hermesc prints the string id for DefineOwnById and the (display + // truncated) text for DefineOwnByIdLong, which a base compile picks as + // soon as the inherited string table pushes the id past 16 bits. Both + // renderings fold to one operand; the raw audit compares the name itself. expect( normalizeDisassemblyLine(' DefineOwnById r7, r8, 2, 11591', strings), - ).toBe(' DefineOwnById r7, r8, 2, "foo"'); + ).toBe(' DefineOwnById r7, r8, 2, '); + expect( + normalizeDisassemblyLine( + ' DefineOwnByIdLong r7, r8, 2, "equivalenceCheckP"...', + strings, + ), + ).toBe(' DefineOwnById r7, r8, 2, '); + expect(() => + normalizeDisassemblyLine(' DefineOwnById r7, r8, 2, 4242', strings), + ).toThrow('unresolved string id 4242'); expect( normalizeDisassemblyLine('Offset in debug table: source 0x0000', strings), ).toBeNull(); @@ -831,10 +844,7 @@ describe('publish metadata never sends JSON null', () => { }); /** the regex-per-line implementation the fast path replaced; must agree */ -function legacyNormalize( - line: string, - strings: Map, -): string | null { +function legacyNormalize(line: string): string | null { if (/^Offset in debug table/.test(line)) return null; let m = /^(\s*New(?:Array|Object)WithBuffer)(?:Long)?(AndParent)?\s+(r\d+)(?:, (r\d+))?(.*)$/.exec( @@ -847,8 +857,8 @@ function legacyNormalize( } m = /^(\s*J[A-Za-z]+?)(Long)?\s+(L\d+|\d+)(.*)$/.exec(line); if (m) return `${m[1]} ${m[3]}${m[4]}`; - m = /^(\s*DefineOwnById\w*\s+r\d+, r\d+, \d+, )(\d+)$/.exec(line); - if (m) line = `${m[1]}"${strings.get(Number(m[2])) ?? `?${m[2]}`}"`; + m = /^(\s*DefineOwnById\w*\s+r\d+, r\d+, \d+, )(\d+)?.*$/.exec(line); + if (m) line = `${m[1]}`; m = /^(\s*)([A-Za-z]+?)(?:LongIndex|Long|Short)?(\s+.*|)$/.exec(line); if (m) line = `${m[1]}${m[2]}${m[3].replace(/\s+/g, ' ')}`; m = /^(\s*StringSwitchImm r\d+, \d+, )\d+(, L\d+, \d+)$/.exec(line); @@ -882,6 +892,7 @@ describe('normalizeDisassemblyLine fast path', () => { ' DefineOwnById r0, r1, 1, 3', ' DefineOwnByIdLong r0, r1, 1, 42', ' DefineOwnByIdShort r0, r1, 1, 7', + ' DefineOwnByIdLong r0, r1, 1, "a-name-past-the-p"...', ' GetByIdShort r1, r0, 1, "foo"', ' GetById r1, r0, 1, "foo"', ' GetByIdLong r1, r0, 1, "foo"', @@ -928,7 +939,7 @@ describe('normalizeDisassemblyLine fast path', () => { ]; for (const line of corpus) { expect(normalizeDisassemblyLine(line, strings)).toBe( - legacyNormalize(line, strings), + legacyNormalize(line), ); } }); @@ -1124,7 +1135,13 @@ describe.if(os.platform() !== 'win32')( expect(result.functions).toBe(0); }); - test('a resolved property name that differs is a difference, wherever the ids point', async () => { + test('a DefineOwnById property name is left to the raw audit, never to the text', async () => { + // hermesc prints this operand as a string id in one build and as + // display-truncated text in the other, so the pretty pass folds it + // away. A name that differs is caught by the raw audit against the + // binary string table; here the fake compiler has no readable HBC, so + // the pass ends unverifiable -- the base is dropped either way and the + // text alone never reports 'equivalent'. const wrong = DELTA_DUMP.replace( 'DefineOwnByIdLong r1, r0, 1, 4', 'DefineOwnByIdLong r1, r0, 1, 3', @@ -1134,9 +1151,9 @@ describe.if(os.platform() !== 'win32')( write('delta.hbc', wrong), write('plain.hbc', PLAIN_DUMP), ); - expect(result.status).toBe('different'); + expect(result.status).toBe('dump-failed'); expect(result.detail).toBe( - 'Function(1 params, 3 registers, 0 symbols): +4: DefineOwnById r1, r0, 1, "baz" vs DefineOwnById r1, r0, 1, "foo"', + 'unsupported or unreadable HBC layout; text-only comparison cannot verify equivalence', ); }); diff --git a/tests/hermes-fuzz-cli.test.ts b/tests/hermes-fuzz-cli.test.ts new file mode 100644 index 0000000..5358193 --- /dev/null +++ b/tests/hermes-fuzz-cli.test.ts @@ -0,0 +1,118 @@ +import { describe, expect, test } from 'bun:test'; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +const script = path.resolve(__dirname, '../scripts/fuzz-hermes-base.ts'); + +describe.skipIf(os.platform() === 'win32')( + 'Hermes fuzz CLI coverage gate', + () => { + test('all compiler failures return nonzero and preserve their inputs', () => { + const dir = mkdtempSync(path.join(os.tmpdir(), 'rnu-fuzz-failure-')); + try { + const compiler = path.join(dir, 'hermesc'); + const out = path.join(dir, 'cases'); + const calls = path.join(dir, 'compiler-called'); + writeFileSync( + compiler, + '#!/bin/sh\n: > "$HERMES_TEST_COMPILER_CALLS"\necho deliberate compiler failure >&2\nexit 1\n', + { mode: 0o755 }, + ); + const result = spawnSync( + process.execPath, + [script, '--rounds', '2', '--seed', '7', '--out', out], + { + env: { + ...process.env, + HERMESC: compiler, + HERMES_TEST_COMPILER_CALLS: calls, + }, + encoding: 'utf8', + timeout: 10_000, + }, + ); + expect(result.error).toBeUndefined(); + expect(result.status).toBe(1); + expect(result.stdout).toContain('equivalent: 0'); + expect(result.stdout).toContain('compile errors (generator): 2'); + expect(existsSync(calls)).toBe(true); + expect( + existsSync(path.join(out, 'round-0000', 'compile-error.txt')), + ).toBe(true); + expect( + existsSync(path.join(out, 'round-0001', 'compile-error.txt')), + ).toBe(true); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, 15_000); + + test.each([ + { args: ['--rounds'] }, + { args: ['--rounds', '--verbose'] }, + { args: ['--rounds', '--seed', '7'] }, + { args: ['--rounds', ''] }, + ])( + 'missing round values fail before starting any work: %j', + ({ args }) => { + const dir = mkdtempSync(path.join(os.tmpdir(), 'rnu-fuzz-args-')); + try { + const compiler = path.join(dir, 'hermesc'); + const out = path.join(dir, 'cases'); + const calls = path.join(dir, 'compiler-called'); + writeFileSync( + compiler, + '#!/bin/sh\n: > "$HERMES_TEST_COMPILER_CALLS"\nexit 1\n', + { mode: 0o755 }, + ); + const result = spawnSync( + process.execPath, + [script, '--out', out, ...args], + { + env: { + ...process.env, + HERMESC: compiler, + HERMES_TEST_COMPILER_CALLS: calls, + }, + encoding: 'utf8', + timeout: 10_000, + }, + ); + expect(result.error).toBeUndefined(); + expect(result.status).toBe(2); + expect(result.stderr).toContain( + '--rounds must be a positive safe integer', + ); + expect(result.stdout).not.toContain('fuzz-hermes-base:'); + expect(existsSync(calls)).toBe(false); + expect(existsSync(out)).toBe(false); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, + 15_000, + ); + + test.each(['0', '-1', '0.5', 'NaN'])( + 'invalid round count %s cannot produce a green empty run', + (rounds) => { + const result = spawnSync( + process.execPath, + [script, '--rounds', rounds], + { + encoding: 'utf8', + timeout: 10_000, + }, + ); + expect(result.error).toBeUndefined(); + expect(result.status).toBe(2); + expect(result.stderr).toContain( + '--rounds must be a positive safe integer', + ); + }, + 15_000, + ); + }, +); diff --git a/tests/hermes-fuzz-literals.test.ts b/tests/hermes-fuzz-literals.test.ts new file mode 100644 index 0000000..e434897 --- /dev/null +++ b/tests/hermes-fuzz-literals.test.ts @@ -0,0 +1,46 @@ +import { describe, expect, test } from 'bun:test'; +import { parse } from '@babel/parser'; +import { fuzzStringLiterals } from '../scripts/hermes-fuzz-literals'; + +describe('Hermes fuzz mutations use actual string boundaries', () => { + test('escaped quotes cannot turn a gap into an invalid property name', () => { + // Minimized from HBC 96, seed 96, round 41. The old regex matched + // `", b: 1, "` and inserted ~ outside the following property-name string. + const source = String.raw`var o = {a: "text\n\t\"quoted\"", b: 1, "flag-beta.map": "value"};`; + const literals = fuzzStringLiterals(source); + expect(literals.map((literal) => literal.value)).toEqual([ + 'text\n\t"quoted"', + 'flag-beta.map', + 'value', + ]); + for (const literal of literals) { + const changed = + source.slice(0, literal.start) + + JSON.stringify(`${literal.value}~`) + + source.slice(literal.end); + expect(() => parse(changed)).not.toThrow(); + } + }); + + test('ignores quotes in comments, regexps and template text', () => { + const source = [ + '// "comment"', + String.raw`const pattern = /["'\d]/;`, + // biome-ignore lint/suspicious/noTemplateCurlyInString: JS source fixture. + 'const template = `text "raw" ${"actual"}`;', + ].join('\n'); + const values = fuzzStringLiterals(source).map((literal) => literal.value); + expect(values).toEqual(['actual']); + }); + + test('retains UTF-16 source offsets and decodes escaped values', () => { + // Use a cooked string: Bun may escape non-ASCII in a tagged raw template. + const source = 'var x = "😀"; var y = "\\u4e2d";'; + const literals = fuzzStringLiterals(source); + expect(literals.map((literal) => literal.value)).toEqual(['😀', '中']); + const spellings = literals.map(({ start, end }) => + source.slice(start, end), + ); + expect(spellings).toEqual(['"😀"', '"\\u4e2d"']); + }); +}); diff --git a/tests/hermes-raw.test.ts b/tests/hermes-raw.test.ts index c835c3f..9245250 100644 --- a/tests/hermes-raw.test.ts +++ b/tests/hermes-raw.test.ts @@ -236,6 +236,54 @@ describe.if(hasHermesc)('lossless Hermes operand audit (real compiler)', () => { ); }); + // hermesc annotates the string operand of DefineOwnByIdLong but not of + // DefineOwnById, so the same instruction prints the text in one build and a + // bare id in the other -- and pretty output cuts that text to a display + // budget. A base whose string table spills past 16 bits makes the delta + // build take the Long form, which used to read as a difference and threw + // away a good base for any property name longer than the budget. + // v96 and older print the text for both widths of PutNewOwnById, so only + // v98's DefineOwnById carries the asymmetry. + test.skipIf(!hasHermesc || probeHbcVersion(hermesc!) !== 98)( + 'a wide DefineOwnById against a foreign base is not a difference', + async () => { + const base = compile( + 'wide-base', + Array.from( + { length: 70000 }, + (_, i) => `globalThis.s${i} = "base string ${i}";`, + ).join('\n'), + ); + // longer than hermesc's display budget, so the Long form prints a cut + // name where the short form prints the id + const name = 'equivalenceCheckPropertyName'; + const source = `globalThis.h = function h(s, v){ return {...s, ${name}: v, b: 1}; };`; + const plain = compile('wide-plain', source); + const delta = compile('wide-delta', source, base); + const pretty = (file: string) => + dump(file, true) + .split('\n') + .filter((line) => line.includes('DefineOwnById')); + // the renderings really are the two the fold has to bridge + expect(pretty(delta)[0]).toContain('DefineOwnByIdLong'); + expect(pretty(delta)[0]).toContain(`"${name.slice(0, 17)}"...`); + expect(pretty(plain)[0]).not.toContain(name.slice(0, 17)); + const result = await compareHermesBytecode(hermesc!, delta, plain); + expect(result.status, result.detail).toBe('equivalent'); + }, + 30_000, + ); + + test('a property name past the pretty limit still has to match', async () => { + const object = (name: string) => + `globalThis.h = function h(s, v){ return {...s, ${name}: v}; };`; + const a = compile('name-a', object('equivalenceCheckPropertyNameAlpha')); + const b = compile('name-b', object('equivalenceCheckPropertyNameBeta')); + const result = await compareHermesBytecode(hermesc!, a, b); + expect(result.status).toBe('different'); + expect(result.detail).toContain('raw instruction'); + }); + test('overflow function headers retain the same runtime fields', async () => { const base = compile('header-base', 'globalThis.old = "older strings";'); const params = Array.from({ length: 140 }, (_, i) => `p${i}`).join(','); diff --git a/tests/hermes-review-regressions.test.ts b/tests/hermes-review-regressions.test.ts new file mode 100644 index 0000000..75876e8 --- /dev/null +++ b/tests/hermes-review-regressions.test.ts @@ -0,0 +1,213 @@ +import { describe, expect, test } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { + type HermesFuzzSummary, + hermesFuzzSucceeded, +} from '../scripts/hermes-fuzz-result'; +import { + compareHermesBytecode, + normalizeDisassemblyLine, +} from '../src/utils/hermes-base'; +import { + type LiteralBuffers, + LiteralResolver, +} from '../src/utils/hermes-literals'; +import { + type HermesSemanticData, + normalizeRawHermesFunction, +} from '../src/utils/hermes-raw'; + +// Synthetic binary sections, not an HBC file emitted by a compiler. The opcode +// name comes from the raw dump; operand bytes are checked by the production audit. +function cachedObject(shapeIndex: number, names = ['field']) { + const strings = new Map(names.map((name, i) => [i + 1, name])); + const objectKeys = Buffer.alloc(1 + names.length * 2); + objectKeys[0] = 0x50 | names.length; + names.forEach((_name, i) => { + objectKeys.writeUInt16LE(i + 1, 1 + i * 2); + }); + const shapes = Buffer.alloc((shapeIndex + 1) * 8); + shapes.writeUInt32LE(names.length, shapeIndex * 8 + 4); + const buffers: LiteralBuffers = { + layout: 'shaped', + version: 98, + values: Buffer.alloc(0), + objectKeys, + shapes, + }; + const bytes = Buffer.alloc(8); + bytes[2] = 1; + bytes.writeUInt32LE(shapeIndex, 3); + const data: HermesSemanticData = { + bytes, + version: 98, + strings, + functions: [{ offset: 0, size: bytes.length, metadata: '[]' }], + bigints: [], + regexps: [], + metadata: '[]', + }; + const resolver = new LiteralResolver(buffers, strings); + const line = ` CacheNewObject r0, r1, ${shapeIndex}, 0`; + const raw = () => + normalizeRawHermesFunction( + [ + `[@ 0] CacheNewObject 0, 1, ${shapeIndex}, 0`, + ], + data, + 0, + buffers, + ); + const pretty = (text = line) => + normalizeDisassemblyLine(text, strings, resolver); + return { data, buffers, strings, resolver, line, raw, pretty }; +} + +describe('CacheNewObject shape references', () => { + test('relocated shapes agree in both pretty and raw comparisons', () => { + const plain = cachedObject(0); + const delta = cachedObject(7); + expect(plain.raw()).toEqual(delta.raw()); + expect(plain.pretty()).toBe(delta.pretty()); + expect(plain.pretty()).toContain('field'); + }); + + test.each([ + [['field'], ['differentField']], + [ + ['first', 'second'], + ['second', 'first'], + ], + [['a b'], ['a b']], + [['sharedLongPropertyPrefix甲'], ['sharedLongPropertyPrefix乙']], + ])('changed keys remain different: %j versus %j', (left, right) => { + const a = cachedObject(0, left); + const b = cachedObject(7, right); + expect(a.pretty()).not.toBe(b.pretty()); + expect(a.raw()).not.toEqual(b.raw()); + }); + + test.each([ + ' CacheNewObject r2, r1, 0, 0', + ' CacheNewObject r0, r2, 0, 0', + ' CacheNewObject r0, r1, 0, 1', + ])('retains registers and the cache operand: %s', (line) => { + const fixture = cachedObject(0); + expect(fixture.pretty(line)).not.toBe(fixture.pretty()); + }); + + test('accepts column padding and tabs without changing key whitespace', () => { + const fixture = cachedObject(0, ['a b']); + expect(fixture.pretty(' CacheNewObject\tr0,\tr1, 0,\t0')).toBe( + fixture.pretty(), + ); + }); + + test('missing shapes and missing keys fail closed', () => { + const fixture = cachedObject(0); + expect(() => fixture.pretty(' CacheNewObject r0, r1, 9, 0')).toThrow( + 'undecodable cached object shape', + ); + fixture.buffers.objectKeys.fill(0); + expect(() => fixture.pretty()).toThrow('undecodable cached object shape'); + }); + + test('unresolved string references still fail closed', () => { + const fixture = cachedObject(0); + fixture.strings.clear(); + expect(() => fixture.pretty()).toThrow('unresolved string id 1'); + }); + + test.each([ + ' CacheNewObject r0, r1, 0', + ' CacheNewObject r0, r1, -1, 0', + ' CacheNewObject r0, r1, 0, 0, 9', + ])('does not silently fold malformed operands: %s', (line) => { + expect(() => cachedObject(0).pretty(line)).toThrow( + 'unsupported cached object operands', + ); + }); + + test('the classic split layout cannot resolve a cached-object shape', () => { + const resolver = new LiteralResolver( + { + layout: 'split', + version: 96, + array: Buffer.alloc(0), + objectKeys: Buffer.alloc(0), + objectValues: Buffer.alloc(0), + }, + new Map(), + ); + expect(() => + normalizeDisassemblyLine( + ' CacheNewObject r0, r1, 0, 0', + new Map(), + resolver, + ), + ).toThrow('undecodable cached object shape'); + }); + + test.skipIf(os.platform() === 'win32')( + 'matching folded text without binary data is still unverifiable', + async () => { + const dir = mkdtempSync(path.join(os.tmpdir(), 'rnu-cached-shape-')); + try { + const compiler = path.join(dir, 'fake-hermesc'); + writeFileSync( + compiler, + `#!${process.execPath}\nconst fs = require('node:fs');\nprocess.stdout.write(fs.readFileSync(process.argv[process.argv.length - 1], 'utf8'));\n`, + { mode: 0o755 }, + ); + const plain = path.join(dir, 'plain.hbc'); + const delta = path.join(dir, 'delta.hbc'); + const dump = (index: number) => + `Function(1 params, 2 registers):\n CacheNewObject r0, r1, ${index}, 0\n`; + writeFileSync(plain, dump(0)); + writeFileSync(delta, dump(7)); + const result = await compareHermesBytecode(compiler, delta, plain); + expect(result.status).toBe('dump-failed'); + expect(result.detail).toContain('text-only comparison cannot verify'); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, + ); +}); + +describe('Hermes fuzz success requires useful coverage', () => { + const success: HermesFuzzSummary = { + rounds: 50, + equivalent: 50, + different: 0, + dumpFailed: 0, + compileErrors: 0, + planted: 5, + plantedMissed: 0, + plantedCompileErrors: 0, + }; + + test('accepts a fully exercised successful run', () => { + expect(hermesFuzzSucceeded(success)).toBe(true); + }); + + test.each([ + { equivalent: 0, compileErrors: 50, planted: 0 }, + { equivalent: 49, compileErrors: 1 }, + { equivalent: 49 }, + { planted: 0 }, + { plantedMissed: 1 }, + { plantedCompileErrors: 1 }, + { different: 1 }, + { dumpFailed: 1 }, + { rounds: 0, equivalent: 0 }, + { rounds: -1, equivalent: -1 }, + { rounds: 0.5, equivalent: 0.5 }, + { rounds: Number.NaN }, + { rounds: Number.POSITIVE_INFINITY }, + ])('rejects incomplete or invalid results: %j', (override) => { + expect(hermesFuzzSucceeded({ ...success, ...override })).toBe(false); + }); +});