Skip to content

perf(fsst): push down '%suffix' LIKE without decompression - #9728

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fsst-suffix-pushdown
Open

perf(fsst): push down '%suffix' LIKE without decompression#9728
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fsst-suffix-pushdown

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

LikeKind::parse handled prefix% and %needle% but not %suffix, so the kernel decompressed the
whole column for those — even though vortex-duckdb lowers suffix(col, 'x') to LIKE %x.

Takes the TODO's backward-scan route: a forward DFA cannot exit early, since a suffix match is only
decided at the last code. The reverse parsing that needs is local; the module doc has the argument.

Benchmarks

fsst_suffix against fsst_suffix_canonicalize, both in one bench binary so this reproduces from a
clean checkout. divan median of 100 samples, 8 repeats; match rates 0–100%. worst is the
canonicalize arm's fastest sample over the pushdown arm's slowest.

dataset B/row canonicalize pushdown ratio worst
rare 44.3 26.18 µs 4.81 µs 5.4x 4.9x
log 127.8 30.84 µs 7.48 µs 4.1x 3.6x
json 96.6 19.27 µs 4.69 µs 4.1x 3.7x
cb 92.5 23.31 µs 6.52 µs 3.6x 3.5x
path 37.8 8.62 µs 5.60 µs 1.5x 1.4x
urls 35.1 8.29 µs 5.60 µs 1.5x 1.3x
email 21.9 6.96 µs 7.21 µs 0.97x 0.79x

email is a wash, and the noisiest case (6.42–8.75 µs across repeats). Its 9-byte suffix exceeds
FSST's 8-byte symbol cap, so the last symbol never settles the match and every row walks 2–3 tokens,
with only 21.9 B/row to amortise it. Deciding on one symbol would be 4.3x, so this is headroom rather
than a floor — say if you would rather gate on row length than carry it.

Tests

91 → 109. run_like now asserts the kernel returned Some before comparing booleans, since the
fallback returns the same answers: unhooking parse_suffix fails 13 tests. Three more mutations each
fail one new test.

AI assistance

Agentic AI assistance; I derived the parity argument by hand and checked the mutations fail.

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Do you have any benchmarks using real world or synthetic data?

LikeKind::parse handled prefix% and %needle% but not %suffix, so those
patterns fell through FsstMatcher::try_new and the kernel decompressed the
whole column. vortex-duckdb already lowers suffix(col, 'x') to LIKE '%x', so
on one column prefix() and contains() match on the code stream while suffix()
does not.

Adds SuffixMatcher, taking the backward-scan route the module TODO sketched.
A forward DFA cannot exit early, because a suffix match is only decided at the
last code, and it measured slower than decompressing. Walking backward makes
the row's last byte the first one compared, so a non-matching row is usually
rejected on one symbol. The reverse parsing that needs is local: escape
ambiguity propagates only through a run of consecutive ESCAPE_CODE bytes, and
that run's parity decides whether the byte left of a token boundary is an
escaped literal or a symbol code.

Every per-code table is sized to the whole code space rather than to the
symbol count, so a code byte past the symbol table -- which only a corrupt
file produces -- reads padding and answers "no match" instead of indexing out
of bounds. The prefix and contains DFAs already absorb that byte in the
padding of their 256-wide tables.

Also adds fsst_suffix and fsst_suffix_canonicalize bench arms, so the
comparison against decompress-and-compare runs from a clean checkout. Six of
the seven datasets are 1.4-4.9x faster even taking each arm's worst sample;
the email dataset is a wash (0.97x), and is the noisiest case in the set.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Yes — fsst_suffix and fsst_suffix_canonicalize arms are in encodings/fsst/benches/fsst_like.rs,
both in one binary so the comparison runs from a clean checkout; table is in the description. The
forward DFA I first pushed lost on all seven datasets, so this is a backward walk from each row's end
now: six datasets are 1.3–4.9x faster taking each arm's worst sample, and email is a wash (0.97x).

Two things the rewrite also fixed: a code byte past the symbol table used to index out of bounds
instead of answering "no match", and the end-to-end cases were passing through the fallback — they now
assert the kernel pushed down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants