Skip to content

Format: the Iceberg layer moves to lakekit - #186

Draft
olekslitus wants to merge 9 commits into
mainfrom
worktree-lakebed-split
Draft

olekslitus wants to merge 9 commits into
mainfrom
worktree-lakebed-split

Conversation

@olekslitus

@olekslitus olekslitus commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

The WASM reader and the shared writer (Workers) now read and write Iceberg manifests, manifest lists and table metadata through lakekit (D24, spec 13), which lakeglass will use too. lakekit is merged (polythings/lakekit#8, #9, #12) and builds on loko (polythings/loko#5, merged) for its store, clock, and random services; the WASM writer takes the time and random numbers from loko's JsClock and JsRandom.

  • timelake-format: host re-exports lakekit's Host plus a WriteHost trait for the two writer limits; commit runs lakekit's commit with Timelake's parts as CommitHooks (last-value index, batch ids, rollup provenance, head-first reloads); FileStats wraps lakekit's ParquetStats with the footer; a file's time partition maps to lakekit's typed partition tuple (time_partition, partition_values).
  • timelake-wasm: manifests read by lakekit instead of its own Avro code; the JS host implements lakekit's traits. Bundle rebuilt (+0.6%).
  • Engine: unchanged except FileStats access; its manifest-equivalence unit test moved to lakekit, whose interop test runs iceberg-rust and lakekit on one table in turn. tests/shared_writer.rs still checks the engine and the shared writer on one table.
  • Fixed: a JavaScript writer whose clock is behind the table's last update dates its commit at that update, so iceberg-rust can load the table.

Dependencies: lakekit 0.1 and loko 0.2 come from crates.io (published 2026-10-22), with no git sources, so timelake-format can be published (cargo publish --dry-run -p timelake-format passes).

Checks: clippy (workspace and wasm32) and rustfmt clean; cargo test --workspace and the WASM crate's tests pass; the bundle matches its sources; the spec index is current. ty and bench queries failed locally only because they ran with the main checkout's virtualenv (an editable timebench from a newer checkout); no Python changed.

🤖 Generated with Claude Code

olekslitus and others added 2 commits October 20, 2026 17:37
The WASM reader and the shared writer read and write Iceberg manifests, manifest lists and
table metadata through lakebed (github.com/polythings/lakebed, D24), which lakeglass uses too.
Timelake keeps its conventions: the last-value index, batch ids, rollup provenance and head-first
reloads plug into lakebed's commit hooks, and a file's time partition maps to lakebed's typed
partition tuple. The engine keeps iceberg-rust (D13); shared_writer.rs still checks that the two
write one table in turn. A writer whose clock is behind the table's last update now dates its
commit at that update, so the snapshot log stays in order.

Until lakebed 0.1.0 is on crates.io, Cargo.toml patches it to a local checkout; replace the
patch with the published version (or a git revision) before merging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 20, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

File statistics keep a bound or null count only when every row group gives it exactly, and
decimals stored as byte arrays get Iceberg's shortest bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@olekslitus olekslitus changed the title Format: the Iceberg layer moves to lakebed Format: the Iceberg layer moves to lakekit Oct 21, 2026
olekslitus and others added 3 commits October 20, 2026 22:16
The Iceberg layer's project was renamed (polythings/lakekit#9): crates lakekit-core and
lakekit-iceberg, pinned at its rename commit until 0.1.0 is on crates.io.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lakekit split its Host trait into Store (objects) and Env (clock, randomness, waiting)
(polythings/lakekit#12). The shared write path's module is now store, and its limits trait
WriteEnv; the WASM writer's JsStore and the test's EngineStore implement Store, Env, and WriteEnv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lakekit split its Env trait into Clock and Random (polythings/lakekit#12). The write path's limits
are WriteLimits, on their own; the WASM writer's JsStore and the test's EngineStore implement Store,
Clock, Random, and WriteLimits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
olekslitus and others added 3 commits October 21, 2026 16:41
…e WASM writer

lakekit commits now take Services, a store, a clock, and a random source
borrowed separately, and its hooks receive them. Timelake's commit passes its
one store for all three, and its hooks take the random source from Services
for the index file's name. Store errors from loko convert into WriteError.

The WASM writer's JsStore is now only a store: JsServices holds it with
loko's JsClock and JsRandom (the system's clock and generator in native
builds), replacing Timelake's own Date.now, setTimeout, and crypto code. A
host without crypto.getRandomValues is now an error instead of falling back
to Math.random. The engine's shared-writer test store uses loko's Clock and
Random APIs. WASM bundle rebuilt; spec 13 and the changelog updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, polythings/loko#5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lakekit and loko are published, so the [patch.crates-io] sections and the
WASM crate's git dependencies are gone, and timelake-format can be published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant