[common] Normalize row-id IN literals inside Range.toRanges - #9630
Open
LuciferYang wants to merge 1 commit into
Open
[common] Normalize row-id IN literals inside Range.toRanges#9630LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
toRanges merges consecutive ids in one linear scan and Range.and intersects with forward-only pointers, so both need ascending, deduplicated input. RowIdPredicateVisitor passed the IN literals in engine order, so a descending list intersected with a BETWEEN dropped most of its ranges, and DataEvolutionBatchScan then never read those rows. Sort and dedupe inside toRanges and document it. Also stop building an inverted Range for BETWEEN 10 AND 5, and let sortAndMergeOverlap return a mutable empty list, which the OR branch accumulates into.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9629
Range.toRangescompresses row ids with a linear scan that merges consecutive values, so it only works on ascending, deduplicated input, andRange.andintersects with two forward-moving pointers, so it needs the same. Nothing established either precondition:RowIdPredicateVisitorpassed the IN literals through in whatever order the engine produced them. A descending_ROW_ID IN (...)list intersected with aBETWEENtherefore lost most of its ranges, and sinceDataEvolutionBatchScan.withFilterfeeds the result towithRowRanges, the dropped ranges are rows that are never read.The precondition now lives in
toRanges, which sorts its own copy and skips repeated ids, and says so in its javadoc. That is where it belongs: the signature isIterable<Long>, so no caller could guess the requirement, and the method has one production caller to keep in step.Two more problems in the same extraction path, both reachable and both fixed here:
BETWEEN 10 AND 5is legal SQL that matches nothing, but the visitor builtnew Range(10, 5)and broke thefrom <= toinvariant the constructor asserts. It now yields no range.Range.sortAndMergeOverlapreturnedCollections.emptyList()for empty input while the visitor's OR branch accumulates into the list it gets back, so the next child'saddAllthrewUnsupportedOperationException. It now returns a mutable list.(_ROW_ID BETWEEN 1 AND 5 AND _ROW_ID BETWEEN 10 AND 20) OR _ROW_ID = 100reaches that, and the inverted-BETWEEN fix above makes empty children easier to produce, which is why the two go together.Tests
RowIdPredicateVisitorTest.testUnsortedInLiteralsIntersectCorrectlybuilds a descending 21-literal IN list (long enough thatPredicateBuilderkeeps a realInleaf), intersects it withBETWEEN 1 AND 6, and asserts the single expected range; a second case feeds 21 copies of one literal and asserts one range rather than overlapping duplicates.RowIdPredicateVisitorTest.testInvertedBetweenIsEmptyasserts the inverted bounds produce no range, then puts that inverted BETWEEN on the left of an OR to cover the accumulation path.Against the unfixed visitor both tests fail.
mvn -pl paimon-common -Dtest=RowIdPredicateVisitorTest,RangeTest,RowRangeIndexTest teston JDK 8: 43 tests, 0 failures.spotless:checkandcheckstyle:checkon paimon-common are clean.