Skip to content

[lake/iceberg] Support custom Iceberg lake table path - #4253

Open
litiliu wants to merge 2 commits into
apache:mainfrom
litiliu:iceberg-custom-lake-table-path-4251
Open

[lake/iceberg] Support custom Iceberg lake table path#4253
litiliu wants to merge 2 commits into
apache:mainfrom
litiliu:iceberg-custom-lake-table-path-4251

Conversation

@litiliu

@litiliu litiliu commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #4251

Fluss currently requires an Iceberg lake table to use the same database and table name as the Fluss table. This change extends the existing table.datalake.database-name and table.datalake.table-name options to Iceberg, allowing the physical Iceberg table path to differ from the logical Fluss table path.

Paimon already supports configuring a custom physical lake table path through #3523. This change provides the corresponding capability for Iceberg.

Brief change log

  • Allow custom lake database and table names for Iceberg tables.
  • Resolve the mapped Iceberg table path when validating, writing, and committing tiered data.
  • Prevent an existing Iceberg table from being rebound to a different physical table path.
  • Preserve the resolved physical path for Flink $lake tables, including Iceberg metadata tables such as $lake$snapshots and union reads.
  • Keep the existing same-name behavior when the mapping options are not configured.

Tests

  • Added validation coverage for supported and unsupported lake formats.
  • Added Iceberg catalog tests for mapping changes before and after the physical table is created.
  • Added an Iceberg tiering test covering validator, writer, and committer access through a mapped physical table path.
  • Added an Iceberg Flink end-to-end test covering tiering, the physical table mapping, $lake, $lake$snapshots, and streaming union read. Its asynchronous result collection and job cancellation have explicit one-minute deadlines.
  • Ran the targeted Flink catalog unit tests (22 tests passed).
  • Ran FlinkUnionReadPrimaryKeyTableITCase#testUnionReadWithCustomLakeTablePath (passed).
  • Ran TableDescriptorValidationTest, IcebergLakeCatalogTest, and IcebergTieringTest (73 tests passed).

API and Format

No public Java API or storage format changes. The existing table-level lake path options now also support Iceberg.

Documentation

Updated the configuration descriptions for table.datalake.database-name and table.datalake.table-name to include Iceberg.

Generative AI disclosure

  • Yes. OpenAI Codex was used to assist with implementation, review, and testing.

@litiliu
litiliu force-pushed the iceberg-custom-lake-table-path-4251 branch 8 times, most recently from f7d70f1 to 69d0785 Compare September 9, 2026 08:09
@litiliu
litiliu force-pushed the iceberg-custom-lake-table-path-4251 branch from 69d0785 to f01820c Compare September 9, 2026 08:29
@litiliu
litiliu marked this pull request as ready for review September 10, 2026 06:43
@litiliu

litiliu commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

@luoyuxia Please take a look when you have time. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[lake/iceberg] Support custom Iceberg lake table path

1 participant