Repository navigation
Format: the Iceberg layer moves to lakekit - #186
Draft
olekslitus wants to merge 9 commits into
Draft
olekslitus wants to merge 9 commits into
olekslitus wants to merge 9 commits into
Conversation
The WASM reader and the shared writer read and write Iceberg manifests, manifest lists and table metadata through lakebed (github.com/polythings/lakebed, D24), which lakeglass uses too. Timelake keeps its conventions: the last-value index, batch ids, rollup provenance and head-first reloads plug into lakebed's commit hooks, and a file's time partition maps to lakebed's typed partition tuple. The engine keeps iceberg-rust (D13); shared_writer.rs still checks that the two write one table in turn. A writer whose clock is behind the table's last update now dates its commit at that update, so the snapshot log stays in order. Until lakebed 0.1.0 is on crates.io, Cargo.toml patches it to a local checkout; replace the patch with the published version (or a git revision) before merging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
File statistics keep a bound or null count only when every row group gives it exactly, and decimals stored as byte arrays get Iceberg's shortest bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Iceberg layer's project was renamed (polythings/lakekit#9): crates lakekit-core and lakekit-iceberg, pinned at its rename commit until 0.1.0 is on crates.io. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lakekit split its Host trait into Store (objects) and Env (clock, randomness, waiting) (polythings/lakekit#12). The shared write path's module is now store, and its limits trait WriteEnv; the WASM writer's JsStore and the test's EngineStore implement Store, Env, and WriteEnv. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lakekit split its Env trait into Clock and Random (polythings/lakekit#12). The write path's limits are WriteLimits, on their own; the WASM writer's JsStore and the test's EngineStore implement Store, Clock, Random, and WriteLimits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 21, 2026
…e WASM writer lakekit commits now take Services, a store, a clock, and a random source borrowed separately, and its hooks receive them. Timelake's commit passes its one store for all three, and its hooks take the random source from Services for the index file's name. Store errors from loko convert into WriteError. The WASM writer's JsStore is now only a store: JsServices holds it with loko's JsClock and JsRandom (the system's clock and generator in native builds), replacing Timelake's own Date.now, setTimeout, and crypto code. A host without crypto.getRandomValues is now an error instead of falling back to Math.random. The engine's shared-writer test store uses loko's Clock and Random APIs. WASM bundle rebuilt; spec 13 and the changelog updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, polythings/loko#5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lakekit and loko are published, so the [patch.crates-io] sections and the WASM crate's git dependencies are gone, and timelake-format can be published. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The WASM reader and the shared writer (Workers) now read and write Iceberg manifests, manifest lists and table metadata through lakekit (D24, spec 13), which lakeglass will use too. lakekit is merged (polythings/lakekit#8, #9, #12) and builds on loko (polythings/loko#5, merged) for its store, clock, and random services; the WASM writer takes the time and random numbers from loko's
JsClockandJsRandom.timelake-format:hostre-exports lakekit'sHostplus aWriteHosttrait for the two writer limits;commitruns lakekit's commit with Timelake's parts asCommitHooks(last-value index, batch ids, rollup provenance, head-first reloads);FileStatswraps lakekit'sParquetStatswith the footer; a file's time partition maps to lakekit's typed partition tuple (time_partition,partition_values).timelake-wasm: manifests read by lakekit instead of its own Avro code; the JS host implements lakekit's traits. Bundle rebuilt (+0.6%).FileStatsaccess; its manifest-equivalence unit test moved to lakekit, whose interop test runs iceberg-rust and lakekit on one table in turn.tests/shared_writer.rsstill checks the engine and the shared writer on one table.Dependencies: lakekit 0.1 and loko 0.2 come from crates.io (published 2026-10-22), with no git sources, so
timelake-formatcan be published (cargo publish --dry-run -p timelake-formatpasses).Checks: clippy (workspace and wasm32) and rustfmt clean;
cargo test --workspaceand the WASM crate's tests pass; the bundle matches its sources; the spec index is current.tyandbench queriesfailed locally only because they ran with the main checkout's virtualenv (an editabletimebenchfrom a newer checkout); no Python changed.🤖 Generated with Claude Code