Skip to content

[AURON #2527] Populate identity partition values in native Iceberg scans - #2528

Open
lyne7-sc wants to merge 4 commits into
apache:masterfrom
lyne7-sc:fix/iceberg-identity-partition-values
Open

lyne7-sc wants to merge 4 commits into
apache:masterfrom
lyne7-sc:fix/iceberg-identity-partition-values

Conversation

@lyne7-sc

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2527

Rationale for this change

Imported partitioned files can omit identity partition columns. Native Iceberg scans may return NULL for these columns instead of the values available in Iceberg partition metadata.

What changes are included in this PR?

  • Populate identity partition columns from partition metadata by field ID for regular and changelog scans.
  • Fall back to Iceberg when scan tasks require different sets of identity partition columns.
  • Fix the test configuration so reference queries actually disable Auron, and verify the reference and native scan plans.

Are there any user-facing changes?

Native Iceberg scans correctly return identity partition values when the columns are absent from the data files.

How was this patch tested?

Compare Auron with Iceberg's reader using Parquet and ORC files that omit identity partition columns. Cover string, date, decimal and NULL values, projections, filters, metadata columns and changelog queries.

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

If yes, include: Generated-by: OpenAI Codex (GPT-6)

ASF guidance: https://www.apache.org/legal/generative-tooling.html

@lyne7-sc
lyne7-sc marked this pull request as ready for review September 15, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native Iceberg scans return NULL for omitted identity partition columns

1 participant