diff --git a/.changeset/like-redos-linear-matcher.md b/.changeset/like-redos-linear-matcher.md new file mode 100644 index 0000000000..93f5cfff09 --- /dev/null +++ b/.changeset/like-redos-linear-matcher.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db': patch +--- + +Fix ReDoS (CWE-1333) in `like()`/`ilike()`: patterns are now matched with an iterative two-pointer walk instead of being compiled to a RegExp, so crafted patterns with many `%` wildcards can no longer trigger catastrophic backtracking on near-miss values. The matcher preserves SQL wildcard semantics when the input value itself contains `%` or `_` characters. diff --git a/AGENTS.md b/AGENTS.md index 077580c31a..69f63a936d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -403,10 +403,12 @@ Use the [coverage map](docs/contributing/oracle-coverage.md) to find an existing owner and its limits before adding another model. The guide explains testing methods; it does not authorize new product behavior or retire existing laws. -When an oracle has a stable subsystem model, write it as executable subsystem +Write every oracle and generated-history test as executable subsystem documentation. Follow the guide's section on literate oracle files. Keep the contract, model, generated history grammar, production driver, and observations -close enough that a reviewer can compare them directly. +visible and close enough that a reviewer can compare them directly. A focused +regression may remain short, but it does not waive this structure for an oracle +and cannot replace applicable oracle coverage. ### Always Add Tests for Bugs diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index a95f4b280c..a3ba192373 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -32,11 +32,10 @@ git ls-files -co --exclude-standard packages \ The command includes tracked and untracked package files so an in-progress audit cannot hide a new owner. A surface is complete only when each primary -owner does one of these things: - -- states its contract, model, history grammar, production path, and observations - in the executable file, or -- records why that structure would add no useful information to a focused test. +owner states its contract, model, history grammar, production path, and +observations in the executable file. Focused regressions outside that inventory +may remain short, but a file named `oracle` or `property` cannot use that +allowance to omit the literate structure. The pass must not change product behavior. If clearer prose exposes a missing model rule or assertion, strengthen the oracle and run it against unchanged @@ -114,7 +113,7 @@ comment and the current API/architecture contract before extending its model. | SQLite persistence and native hosts | [persisted histories](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/persisted.test.ts), [driver contracts](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/contracts/sqlite-driver-contract.ts), [browser OPFS lifecycle](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-page-lifecycle-oracle.test.ts), [worker diagnostics](https://github.com/TanStack/db/blob/main/packages/browser-db-sqlite-persistence/tests/opfs-worker-diagnostics-oracle.test.ts), [113-law manifest](https://github.com/TanStack/db/blob/main/packages/db-collection-e2e/src/fixtures/persisted-conformance-manifest.ts) | Cache/remote rejection/peer/reopen histories, exact driver results, controlled page/worker ownership, and diagnostic-cause retention. Fake workers and synthetic page events do not prove native handle release or real bfcache admission. The manifest excludes progressive and move suites; registration and shim runs are not device execution. | | Offline execution | [scheduler](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/KeyScheduler.property.test.ts), [leadership](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/leadership-replay.property.test.ts), [settlement](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/transaction-settlement.property.test.ts), [serialization](https://github.com/TanStack/db/blob/main/packages/offline-transactions/tests/transaction-serializer.property.test.ts) | Declarative FIFO eligibility, per-transaction outcomes, durable state and typed wire trees. Issued work may finish after ownership loss, but new work must not start. Exactly-once network execution is not promised. | | Frameworks | [React conformance](https://github.com/TanStack/db/blob/main/packages/react-db/tests/conformance.test.tsx), [React pagination](https://github.com/TanStack/db/blob/main/packages/react-db/tests/infinite-query-conformance.test.tsx), [shared suites](https://github.com/TanStack/db/tree/main/packages/db-collection-e2e/src/suites) | Exact exposed rows/pages and each framework's own lifecycle cuts. A React witness does not prove Vue/Solid/Angular/Svelte scheduling. Preserve their receiving registrations. | -| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update.property.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, and compiled output bags. Hash collision freedom is not promised. Unsupported composite cursors reject. | +| Structural values and ordered primitives | [hash values](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash.property.test.ts), [hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-graph.property.test.ts), [mixed hash graphs](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-mixed-graph.property.test.ts), [hash retry](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/hash-failure-retry.property.test.ts), [comparison](https://github.com/TanStack/db/blob/main/packages/db/tests/comparison.property.test.ts), [deep equality](https://github.com/TanStack/db/blob/main/packages/db/tests/utils.property.test.ts), [cursor](https://github.com/TanStack/db/blob/main/packages/db/tests/cursor.property.test.ts), [indexes](https://github.com/TanStack/db/blob/main/packages/db/tests/index-update.property.test.ts), [query identity](https://github.com/TanStack/db/blob/main/packages/db/tests/query/identity-output-shape-oracle.test.ts), [LIKE semantics](https://github.com/TanStack/db/blob/main/packages/db/tests/query/compiler/evaluators.test.ts) | Independent flat values, graph topology, algebraic laws, Map/group/sort recomputation, expression denotation, LIKE wildcard refinement, and compiled output bags. The LIKE owner covers boolean string matching and bounded work, not nullish three-valued logic or a general Unicode collation contract. Hash collision freedom is not promised. Unsupported composite cursors reject. | | Boundary refinements | [cleanup/restart](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-cleanup-restart-oracle.test.ts), [metadata publication](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-metadata-publication-oracle.property.test.ts), [state retention](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-state-retention-oracle.property.test.ts), [acquisition cells](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-subscription-lifecycle-oracle.test.ts), [D2 source reconciliation](https://github.com/TanStack/db/blob/main/packages/db/tests/d2-source-reconciliation-oracle.property.test.ts), [top-K support windows](https://github.com/TanStack/db/blob/main/packages/db-ivm/tests/operators/topk-support-window-oracle.test.ts), [nested Query work](https://github.com/TanStack/db/blob/main/packages/query-db-collection/tests/includes-work-counter-oracle.test.ts) | Explicit lifecycle products, independent source maps and weighted relations, exact publication cuts, support/multiplicity, and value-plus-work observations. These refine the larger subsystem models; they do not replace them. | | Small structures and test mechanics | [SortedMap](https://github.com/TanStack/db/blob/main/packages/db/tests/SortedMap.test.ts), [cleanup queue](https://github.com/TanStack/db/blob/main/packages/db/tests/cleanup-queue.property.test.ts), [guarded replay](https://github.com/TanStack/db/blob/main/packages/db/tests/oracle-replay.test.ts) | Map/full-sort and appointment-list models with executed target/seed/path checks. Callback-reentrant scheduling is outside the initial cleanup-queue domain. | diff --git a/docs/contributing/oracle-tests.md b/docs/contributing/oracle-tests.md index bea51eac49..69df8606de 100644 --- a/docs/contributing/oracle-tests.md +++ b/docs/contributing/oracle-tests.md @@ -250,9 +250,16 @@ cannot create a named phase, the prose claims more reach than the test has. ### Keep the prose proportional -Do not turn every regression test into an essay. Use this form when a file owns -a reusable law, state machine, lifecycle boundary, or reference model. A focused -test can remain short when its name and setup already state the whole contract. +Every oracle and generated-history test must make the five layers visible. This +structure is not optional when the file owns a reusable law, state machine, +lifecycle boundary, or reference model. A focused regression that is not an +oracle can remain short when its name and setup already state the whole contract, +but it does not replace applicable oracle coverage. + +Mandatory structure does not mean five classes or a long essay. A compact +oracle can state its contract and limits in one opening comment, keep a pure +model beside it, define a small grammar, drive the production entry point, and +compare the promised observations at a named checkpoint. As a starting budget, add only prose that helps a reader answer one of the five layer questions. After the first draft, remove comments that only translate the diff --git a/packages/db/src/query/compiler/evaluators.ts b/packages/db/src/query/compiler/evaluators.ts index 55b8c68f49..5e1c317863 100644 --- a/packages/db/src/query/compiler/evaluators.ts +++ b/packages/db/src/query/compiler/evaluators.ts @@ -635,6 +635,13 @@ export function isCaseWhenConditionTrue(value: any): boolean { /** * Evaluates LIKE/ILIKE patterns + * + * `%` matches any sequence of characters (including none), `_` matches + * exactly one. The pattern is matched with an iterative two-pointer walk + * instead of being compiled to a RegExp: patterns with many `%` wildcards + * would produce overlapping `.*` segments whose catastrophic backtracking + * makes near-miss inputs take exponential time (CWE-1333). The walk is + * O(value.length * pattern.length) in the worst case. */ function evaluateLike( value: any, @@ -648,15 +655,37 @@ function evaluateLike( const searchValue = caseInsensitive ? value.toLowerCase() : value const searchPattern = caseInsensitive ? pattern.toLowerCase() : pattern - // Convert SQL LIKE pattern to regex - // First escape all regex special chars except % and _ - let regexPattern = searchPattern.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) - - // Then convert SQL wildcards to regex - regexPattern = regexPattern.replace(/%/g, `.*`) // % matches any sequence - regexPattern = regexPattern.replace(/_/g, `.`) // _ matches any single char + let valueIndex = 0 + let patternIndex = 0 + // Position of the most recent `%` and the value position it restarts from + let starPatternIndex = -1 + let starValueIndex = 0 + + while (valueIndex < searchValue.length) { + const patternChar = + patternIndex < searchPattern.length + ? searchPattern[patternIndex] + : undefined + if (patternChar === `%`) { + starPatternIndex = patternIndex + starValueIndex = valueIndex + patternIndex++ + } else if (patternChar === `_` || patternChar === searchValue[valueIndex]) { + valueIndex++ + patternIndex++ + } else if (starPatternIndex !== -1) { + // Mismatch after a `%`: let it consume one more character and retry + starValueIndex++ + valueIndex = starValueIndex + patternIndex = starPatternIndex + 1 + } else { + return false + } + } - // 's' (dotAll flag) makes '.' match all characters including line terminations - const regex = new RegExp(`^${regexPattern}$`, 's') - return regex.test(searchValue) + // The value is consumed; only trailing `%` wildcards may remain + while (searchPattern[patternIndex] === `%`) { + patternIndex++ + } + return patternIndex === searchPattern.length } diff --git a/packages/db/tests/query/compiler/evaluators.test.ts b/packages/db/tests/query/compiler/evaluators.test.ts index dac457867a..85f8bdf241 100644 --- a/packages/db/tests/query/compiler/evaluators.test.ts +++ b/packages/db/tests/query/compiler/evaluators.test.ts @@ -4,6 +4,172 @@ import { compileExpression } from '../../../src/query/compiler/evaluators.js' import { Func, PropRef, Value } from '../../../src/query/ir.js' import type { NamespacedRow } from '../../../src/types.js' +/** + * # Does the LIKE evaluator preserve wildcard meaning without regex work? + * + * Contract: `like` and `ilike` match the complete string. `%` consumes zero or + * more UTF-16 code units, `_` consumes exactly one, and every other code unit + * is literal. `ilike` applies the evaluator's established lowercase fold. + * + * Model: `referenceLike` uses a dynamic program, not production's two-pointer + * walk. It predicts only the boolean result for non-null strings. + * + * History grammar: each case contains a value, a pattern, and a case-fold flag. + * The small alphabet concentrates on literal/wildcard precedence, `_`, case, + * newlines, and non-ASCII code units. Every generated string is legal. Pinned + * witnesses reconstruct the reported `%` collision; removing `%` removes that + * distinction, while empty and literal-only cases exercise its boundaries. + * + * Production driver: compile a public `like` or `ilike` expression and invoke + * it at the synchronous evaluator boundary. + * + * Refinement check: production and the model must return the same boolean for + * every fixed, random, or replayed case. The recorder retains the seed and the + * first five mismatches. A deliberate wrong-result control proves that the + * comparison rejects the literal-before-wildcard regression. Nullish SQL + * three-valued logic and broader Unicode folding remain in focused tests. + */ +type LikeCase = { + value: string + pattern: string + caseInsensitive: boolean +} + +type LikeMismatch = LikeCase & { + expected: boolean + actual: boolean +} + +const FIXED_LIKE_ORACLE_SEED = 0x1745 +const likeReplaySeedText = process.env.TANSTACK_DB_LIKE_ORACLE_SEED +const likeReplaySeed = + likeReplaySeedText === undefined ? undefined : Number(likeReplaySeedText) +const LIKE_ORACLE_RUNS = Number( + process.env.TANSTACK_DB_LIKE_ORACLE_RUNS ?? 10_000, +) +const likeCalibrationCases: Array = [ + { value: ``, pattern: ``, caseInsensitive: false }, + { value: `A`, pattern: `a`, caseInsensitive: true }, + { value: `anything`, pattern: `%`, caseInsensitive: false }, + { value: `a`, pattern: `_`, caseInsensitive: true }, + { value: `100% done`, pattern: `100%done`, caseInsensitive: false }, +] + +if ( + likeReplaySeedText !== undefined && + (likeReplaySeedText.trim() === `` || + !Number.isSafeInteger(likeReplaySeed) || + likeReplaySeed! < 0 || + likeReplaySeed! > 0xffff_ffff) +) { + throw new Error(`TANSTACK_DB_LIKE_ORACLE_SEED must be a uint32 integer`) +} +if ( + !Number.isSafeInteger(LIKE_ORACLE_RUNS) || + LIKE_ORACLE_RUNS < likeCalibrationCases.length +) { + throw new Error( + `TANSTACK_DB_LIKE_ORACLE_RUNS must be an integer of at least ${likeCalibrationCases.length}`, + ) +} + +const likeOracleCampaigns = + likeReplaySeed === undefined + ? [ + { name: `fixed`, seed: FIXED_LIKE_ORACLE_SEED }, + { name: `random`, seed: Math.floor(Math.random() * 0x1_0000_0000) }, + ] + : [{ name: `replay`, seed: likeReplaySeed }] + +function referenceLike( + value: string, + pattern: string, + caseInsensitive: boolean, +): boolean { + const searchValue = caseInsensitive ? value.toLowerCase() : value + const searchPattern = caseInsensitive ? pattern.toLowerCase() : pattern + let previous = new Array(searchPattern.length + 1).fill(false) + previous[0] = true + + for ( + let patternIndex = 1; + patternIndex <= searchPattern.length; + patternIndex++ + ) { + previous[patternIndex] = + searchPattern[patternIndex - 1] === `%` && previous[patternIndex - 1]! + } + + for (let valueIndex = 1; valueIndex <= searchValue.length; valueIndex++) { + const current = new Array(searchPattern.length + 1).fill(false) + for ( + let patternIndex = 1; + patternIndex <= searchPattern.length; + patternIndex++ + ) { + const patternCharacter = searchPattern[patternIndex - 1] + current[patternIndex] = + patternCharacter === `%` + ? current[patternIndex - 1]! || previous[patternIndex]! + : (patternCharacter === `_` || + patternCharacter === searchValue[valueIndex - 1]) && + previous[patternIndex - 1]! + } + previous = current + } + + return previous[searchPattern.length]! +} + +function* seededLikeCases(count: number, seed: number): Generator { + // Repeated wildcard entries keep the bounded campaign concentrated on the + // precedence boundary while retaining literals, case folds, and newlines. + const alphabet = [`a`, `b`, `A`, `B`, `%`, `%`, `%`, `_`, `_`, `.`, `\n`, `é`] + let state = seed + const next = () => { + state = (Math.imul(state, 1664525) + 1013904223) >>> 0 + return state + } + const pick = (exclusiveUpperBound: number) => + Math.floor((next() / 0x1_0000_0000) * exclusiveUpperBound) + const build = (length: number) => { + let result = `` + for (let index = 0; index < length; index++) { + result += alphabet[pick(alphabet.length)] + } + return result + } + + yield* likeCalibrationCases + for (let index = likeCalibrationCases.length; index < count; index++) { + const value = build(pick(8)) + const pattern = build(pick(8)) + yield { value, pattern, caseInsensitive: pick(2) === 1 } + } +} + +function evaluateLikeWithProduction(testCase: LikeCase): boolean { + const functionName = testCase.caseInsensitive ? `ilike` : `like` + return compileExpression( + new Func(functionName, [ + new Value(testCase.value), + new Value(testCase.pattern), + ]), + )({}) +} + +function findLikeMismatch( + testCase: LikeCase, + actual: boolean, +): LikeMismatch | undefined { + const expected = referenceLike( + testCase.value, + testCase.pattern, + testCase.caseInsensitive, + ) + return actual === expected ? undefined : { ...testCase, expected, actual } +} + describe(`evaluators`, () => { describe(`compileExpression`, () => { it(`handles unknown expression type`, () => { @@ -316,6 +482,150 @@ describe(`evaluators`, () => { expect(compiled({})).toBe(true) }) + it(`handles like with wildcard in the middle`, () => { + const func = new Func(`like`, [ + new Value(`hello brave new world`), + new Value(`hello%world`), + ]) + const compiled = compileExpression(func) + + expect(compiled({})).toBe(true) + }) + + it(`treats % as a wildcard when the value also contains %`, () => { + const likeFunc = compileExpression( + new Func(`like`, [new Value(`100% done`), new Value(`100%done`)]), + ) + const ilikeFunc = compileExpression( + new Func(`ilike`, [ + new Value(`A% LONG VALUE`), + new Value(`a%value`), + ]), + ) + + expect(likeFunc({})).toBe(true) + expect(ilikeFunc({})).toBe(true) + }) + + it(`rejects the literal-before-wildcard wrong result`, () => { + const witness: LikeCase = { + value: `100% done`, + pattern: `100%done`, + caseInsensitive: false, + } + + expect( + findLikeMismatch(witness, evaluateLikeWithProduction(witness)), + ).toBeUndefined() + expect(findLikeMismatch(witness, false)).toEqual({ + ...witness, + expected: true, + actual: false, + }) + }) + + it.each(likeOracleCampaigns)( + `matches the LIKE law in the $name campaign`, + ({ seed }) => { + let mismatchCount = 0 + const mismatchSamples: Array = [] + const reach = { + caseSensitive: 0, + caseInsensitive: 0, + percentPattern: 0, + underscorePattern: 0, + literalPattern: 0, + } + + for (const testCase of seededLikeCases(LIKE_ORACLE_RUNS, seed)) { + reach[ + testCase.caseInsensitive ? `caseInsensitive` : `caseSensitive` + ]++ + if (testCase.pattern.includes(`%`)) reach.percentPattern++ + if (testCase.pattern.includes(`_`)) reach.underscorePattern++ + if (/[^%_]/u.test(testCase.pattern)) reach.literalPattern++ + + const mismatch = findLikeMismatch( + testCase, + evaluateLikeWithProduction(testCase), + ) + if (mismatch !== undefined) { + mismatchCount++ + if (mismatchSamples.length < 5) { + mismatchSamples.push(mismatch) + } + } + } + + expect({ seed, mismatchCount, mismatchSamples }).toEqual({ + seed, + mismatchCount: 0, + mismatchSamples: [], + }) + expect( + Object.entries(reach).filter(([, count]) => count === 0), + ).toEqual([]) + }, + ) + + it(`handles like where _ must match exactly one character`, () => { + const func = new Func(`like`, [new Value(`hell`), new Value(`hell_`)]) + const compiled = compileExpression(func) + + expect(compiled({})).toBe(false) + }) + + it(`handles like with a pattern of only wildcards`, () => { + const func = new Func(`like`, [new Value(``), new Value(`%%`)]) + const compiled = compileExpression(func) + + expect(compiled({})).toBe(true) + }) + + it(`handles like with an empty pattern`, () => { + const emptyValue = compileExpression( + new Func(`like`, [new Value(``), new Value(``)]), + ) + const nonEmptyValue = compileExpression( + new Func(`like`, [new Value(`a`), new Value(``)]), + ) + + expect(emptyValue({})).toBe(true) + expect(nonEmptyValue({})).toBe(false) + }) + + it(`handles like matching across line breaks`, () => { + const func = new Func(`like`, [ + new Value(`hello\nworld`), + new Value(`hello%world`), + ]) + const compiled = compileExpression(func) + + expect(compiled({})).toBe(true) + }) + + it(`evaluates pathological wildcard patterns without backtracking (ReDoS)`, () => { + // Compiled to a regex, this pattern produces 20 overlapping `.*` + // segments; the near-miss value (fails only at the last character) + // then made the regex engine backtrack exponentially and hang. + const pattern = `a%`.repeat(19) + `a` + const nearMiss = `a`.repeat(200) + `b` + const likeFunc = compileExpression( + new Func(`like`, [new Value(nearMiss), new Value(pattern)]), + ) + const ilikeFunc = compileExpression( + new Func(`ilike`, [ + new Value(nearMiss.toUpperCase()), + new Value(pattern), + ]), + ) + + const start = performance.now() + expect(likeFunc({})).toBe(false) + expect(ilikeFunc({})).toBe(false) + expect(performance.now() - start).toBeLessThan(1000) + }) + it(`handles like with null value (3-valued logic)`, () => { const func = new Func(`like`, [new Value(null), new Value(`hello%`)]) const compiled = compileExpression(func)