Skip to content

[common] Default missing variant shredding fields to the unshredded index - #9616

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/variant-rowreader-unshredded-npe
Open

[common] Default missing variant shredding fields to the unshredded index#9616
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/variant-rowreader-unshredded-npe

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #9615

BaseVariantReader.RowReader's constructor unboxed a map lookup straight into an int:

fieldInputIndices[i] =
        schema.objectSchemaMap != null
                ? schema.objectSchemaMap.get(targetFields.get(i).name())
                : -1;

objectSchemaMap holds only the fields the file actually shredded, so reading a struct whose fields are not all shredded threw a NullPointerException while the reader was being built, taking the whole scan with it. Shredding is inferred per data file by default, so the same query can work on one file and fail on the next.

getOrDefault(name, -1) is all it takes, because -1 is the convention the rest of the class already speaks. The field's own comment says "or -1 if it doesn't exist in object typed_value", VariantSchema documents the same thing for its indices, and readFromTyped carries the branch that reads such a field out of the untyped value:

} else if (unshreddedObject != null) {
    ...
    GenericVariant unshreddedField = unshreddedObject.getFieldByKey(fieldName);

Since the constructor could never produce -1, that branch and the needUnshreddedObject flag guarding it were both dead code. This makes them reachable, which is the behavior the class was written for.

I checked the other lookups in org.apache.paimon.data.variant while I was there: they all take the result as an Integer and null-check it, or go through containsKey, so this was the only site.

Tests

BaseVariantReaderTest.testRowReaderWithUnshreddedTargetField builds a shredding schema covering only field a, asks for struct<a int, b string>, and reads a row back. b is asserted to come through as "hello", which is only reachable through the unshredded branch, so the test covers the revived path rather than just the absence of the crash. A second case reads {"a": 27}, where the file has no untyped value at all, and expects b to be null.

Against the unfixed reader the test fails with a NullPointerException out of BaseVariantReader.create.

mvn -pl paimon-common -Dtest=BaseVariantReaderTest test on JDK 8: 1 test, 0 failures. spotless:check and checkstyle:check on paimon-common are clean.

…ndex

BaseVariantReader.RowReader's constructor unboxed
schema.objectSchemaMap.get(name) into an int[]. The map only holds
fields the file actually shredded, so reading a struct whose fields are
not all shredded threw a NullPointerException while building the
reader.

Use getOrDefault(name, -1), the convention the class already documents
and the value its unshredded merge branch waits for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Extracting a struct from a variant column NPEs when one target field is not shredded

1 participant