Skip to content

[common] Normalize row-id IN literals inside Range.toRanges - #9630

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/rowid-range-sortedness
Open

[common] Normalize row-id IN literals inside Range.toRanges#9630
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/rowid-range-sortedness

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9629

Range.toRanges compresses row ids with a linear scan that merges consecutive values, so it only works on ascending, deduplicated input, and Range.and intersects with two forward-moving pointers, so it needs the same. Nothing established either precondition: RowIdPredicateVisitor passed the IN literals through in whatever order the engine produced them. A descending _ROW_ID IN (...) list intersected with a BETWEEN therefore lost most of its ranges, and since DataEvolutionBatchScan.withFilter feeds the result to withRowRanges, the dropped ranges are rows that are never read.

The precondition now lives in toRanges, which sorts its own copy and skips repeated ids, and says so in its javadoc. That is where it belongs: the signature is Iterable<Long>, so no caller could guess the requirement, and the method has one production caller to keep in step.

Two more problems in the same extraction path, both reachable and both fixed here:

BETWEEN 10 AND 5 is legal SQL that matches nothing, but the visitor built new Range(10, 5) and broke the from <= to invariant the constructor asserts. It now yields no range.

Range.sortAndMergeOverlap returned Collections.emptyList() for empty input while the visitor's OR branch accumulates into the list it gets back, so the next child's addAll threw UnsupportedOperationException. It now returns a mutable list. (_ROW_ID BETWEEN 1 AND 5 AND _ROW_ID BETWEEN 10 AND 20) OR _ROW_ID = 100 reaches that, and the inverted-BETWEEN fix above makes empty children easier to produce, which is why the two go together.

Tests

RowIdPredicateVisitorTest.testUnsortedInLiteralsIntersectCorrectly builds a descending 21-literal IN list (long enough that PredicateBuilder keeps a real In leaf), intersects it with BETWEEN 1 AND 6, and asserts the single expected range; a second case feeds 21 copies of one literal and asserts one range rather than overlapping duplicates.

RowIdPredicateVisitorTest.testInvertedBetweenIsEmpty asserts the inverted bounds produce no range, then puts that inverted BETWEEN on the left of an OR to cover the accumulation path.

Against the unfixed visitor both tests fail.

mvn -pl paimon-common -Dtest=RowIdPredicateVisitorTest,RangeTest,RowRangeIndexTest test on JDK 8: 43 tests, 0 failures. spotless:check and checkstyle:check on paimon-common are clean.

toRanges merges consecutive ids in one linear scan and Range.and
intersects with forward-only pointers, so both need ascending,
deduplicated input. RowIdPredicateVisitor passed the IN literals in
engine order, so a descending list intersected with a BETWEEN dropped
most of its ranges, and DataEvolutionBatchScan then never read those
rows.

Sort and dedupe inside toRanges and document it. Also stop building an
inverted Range for BETWEEN 10 AND 5, and let sortAndMergeOverlap
return a mutable empty list, which the OR branch accumulates into.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Unsorted _ROW_ID IN literals drop rows from a data-evolution scan

1 participant