Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ jobs:
env:
RUSTFLAGS: -D warnings
run: |
cargo check -p lakebed-iceberg --no-default-features
cargo check -p lakebed-iceberg --features write
cargo check -p lakebed-iceberg --features arrow
cargo check -p lakebed-iceberg --features parquet-stats
cargo check -p lakekit-iceberg --no-default-features
cargo check -p lakekit-iceberg --features write
cargo check -p lakekit-iceberg --features arrow
cargo check -p lakekit-iceberg --features parquet-stats

wasm:
name: WebAssembly (browser and WASI targets)
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Agent instructions

lakebed: table formats (Apache Iceberg first) read and written in Rust without I/O of their own, for WebAssembly and
native targets. One crate per format (`crates/lakebed-<format>`) on a shared I/O boundary (`crates/lakebed-core`).
lakekit: table formats (Apache Iceberg first) read and written in Rust without I/O of their own, for WebAssembly and
native targets. One crate per format (`crates/lakekit-<format>`) on a shared I/O boundary (`crates/lakekit-core`).
Timelake and lakeglass use it.

## Before changing code
Expand Down
18 changes: 12 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,27 @@
# Changelog

All notable changes to lakebed. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
All notable changes to lakekit. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
Comment thread
coderabbitai[bot] marked this conversation as resolved.
versions follow [Semantic Versioning](https://semver.org/).

## Unreleased

### Added

- `lakebed-core`: the `Host` trait for object stores, clocks, and randomness; `MemoryHost`; one `Error` type.
- `lakebed-iceberg`: Iceberg table metadata, schemas, partition specs, manifest lists, and manifests read for format
- `lakekit-core`: the `Host` trait for object stores, clocks, and randomness; `MemoryHost`; one `Error` type.
- `lakekit-iceberg`: Iceberg table metadata, schemas, partition specs, manifest lists, and manifests read for format
versions 1 to 3, including delete files and deletion vector references; values in Iceberg's single-value encoding.
- `lakebed-iceberg` feature `write`: format v2 manifests, manifest lists, and metadata updates, and commits to
- `lakekit-iceberg` feature `write`: format v2 manifests, manifest lists, and metadata updates, and commits to
file-system tables with options (`CommitOptions`) and hooks (`CommitHooks`). A writer whose clock is behind the
table's last update dates its commit at that update, so the snapshot log stays in order.
- `lakebed-iceberg` features `arrow` (schemas as Arrow schemas with field ids) and `parquet-stats` (a data file's
- `lakekit-iceberg` features `arrow` (schemas as Arrow schemas with field ids) and `parquet-stats` (a data file's
Iceberg statistics from its Parquet footer; a bound or null count only when every row group gives it exactly, and
decimals as Iceberg's shortest big-endian bytes).
- `lakebed`: re-exports the format crates behind a feature each.
- `lakekit`: re-exports the format crates behind a feature each.

### Changed

- The project is named lakekit, and its crates `lakekit-core`, `lakekit-iceberg`, and `lakekit`; it was lakebed
(`lakebed-core`, …) until 2026-10-21. Nothing was published under the old names; GitHub redirects the old
repository URL.

The Iceberg code started as Timelake's shared writer (`timelake-format`), made general.
10 changes: 5 additions & 5 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Contributing to lakebed
# Contributing to lakekit

This is the code guide for everyone who changes this repository, people and AI agents alike. Formatters and linters
enforce the mechanical rules (see [Checks](#checks)). This guide covers what they can't: how code should read.
Expand All @@ -8,9 +8,9 @@ enforce the mechanical rules (see [Checks](#checks)). This guide covers what the

1. **Readable first.** Someone new should understand a function from its name, its signature, and one pass through
the body.
2. **One crate per format.** A format's crate depends on `lakebed-core` and on nothing of another format's, so adding
2. **One crate per format.** A format's crate depends on `lakekit-core` and on nothing of another format's, so adding
one never touches the others. Formats are shown as they are: no shared model flattens their differences.
3. **No I/O of our own.** Crates read and write through `lakebed_core::Host`. They never open sockets, files, or
3. **No I/O of our own.** Crates read and write through `lakekit_core::Host`. They never open sockets, files, or
threads, so they build for `wasm32-unknown-unknown`.
4. **One definition of each thing.** Before writing a helper, constant, or type, look for an existing one.
5. **Behaviour, spec, and tests change together.** A change users can see updates its spec, adds or updates tests, and
Expand All @@ -28,7 +28,7 @@ enforce the mechanical rules (see [Checks](#checks)). This guide covers what the

## Credit

lakebed learns from other projects' ideas, never from copying their code. [PRIOR_ART.md](PRIOR_ART.md) lists the
lakekit learns from other projects' ideas, never from copying their code. [PRIOR_ART.md](PRIOR_ART.md) lists the
ideas we took.

- **Never copy code**, whatever its license: not by pasting, and not by translating it into another language.
Expand All @@ -38,7 +38,7 @@ ideas we took.

## Rust

- **Errors:** `lakebed_core::Error`, with the variant that says what a caller can do about it (`NotFound`, `Conflict`,
- **Errors:** `lakekit_core::Error`, with the variant that says what a caller can do about it (`NotFound`, `Conflict`,
`CommitUnknown`, `Unsupported`, `Corrupt`). No `unwrap()` outside tests.
- **Types over strings:** enums and newtypes instead of strings and magic numbers. Parse once at the boundary.
- **Don't trust files.** Check lengths and offsets before using them, cap allocations, and return an error instead of
Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
[workspace]
resolver = "2"
members = ["crates/lakebed-core", "crates/lakebed-iceberg", "crates/lakebed"]
members = ["crates/lakekit-core", "crates/lakekit-iceberg", "crates/lakekit"]

[workspace.package]
version = "0.1.0"
edition = "2024"
rust-version = "1.99"
license = "Apache-2.0"
repository = "https://github.com/polythings/lakebed"
repository = "https://github.com/polythings/lakekit"
keywords = ["lakehouse", "iceberg", "wasm", "table-format"]
categories = ["database-implementations", "wasm"]

[workspace.dependencies]
lakebed-core = { path = "crates/lakebed-core", version = "0.1.0" }
lakebed-iceberg = { path = "crates/lakebed-iceberg", version = "0.1.0", default-features = false }
lakekit-core = { path = "crates/lakekit-core", version = "0.1.0" }
lakekit-iceberg = { path = "crates/lakekit-iceberg", version = "0.1.0", default-features = false }
thiserror = "2"
serde_json = "1"
apache-avro = { version = "0.22", default-features = false }
Expand Down
6 changes: 3 additions & 3 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
lakebed
Copyright 2026 Oleksandr Litus and the lakebed contributors
lakekit
Copyright 2026 Oleksandr Litus and the lakekit contributors

This product is licensed under the Apache License, Version 2.0 (see LICENSE).

Ideas lakebed borrowed from other projects are credited in PRIOR_ART.md.
Ideas lakekit borrowed from other projects are credited in PRIOR_ART.md.
12 changes: 6 additions & 6 deletions PRIOR_ART.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Prior art

lakebed stands on other open-source projects and learns from more. This file credits the ideas we took. We take ideas,
never code: every line of lakebed is our own. How we credit is in [CONTRIBUTING.md](CONTRIBUTING.md#credit).
lakekit stands on other open-source projects and learns from more. This file credits the ideas we took. We take ideas,
never code: every line of lakekit is our own. How we credit is in [CONTRIBUTING.md](CONTRIBUTING.md#credit).

## Foundations

| Project | License | What lakebed uses |
| Project | License | What lakekit uses |
|---|---|---|
| [Apache Iceberg](https://iceberg.apache.org) | Apache-2.0 | The table format and its specification |
| [iceberg-rust](https://github.com/apache/iceberg-rust) | Apache-2.0 | The reference the interop tests check against (a test dependency only) |
Expand All @@ -14,9 +14,9 @@ never code: every line of lakebed is our own. How we credit is in [CONTRIBUTING.

## Ideas

| Idea | From | License of the source | Where in lakebed |
| Idea | From | License of the source | Where in lakekit |
|---|---|---|---|
| A format library with no I/O of its own: the application supplies storage through a trait the library calls | [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs) (its `Engine` trait) | Apache-2.0 | `lakebed-core`'s `Host` |
| A format library with no I/O of its own: the application supplies storage through a trait the library calls | [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs) (its `Engine` trait) | Apache-2.0 | `lakekit-core`'s `Host` |
| One crate per backend under a small shared core, re-exported by an umbrella crate with a feature each | [arrow-rs](https://github.com/apache/arrow-rs), [Apache OpenDAL](https://github.com/apache/opendal) | Apache-2.0 | The workspace layout |
| Commits to tables without a catalog: `v<N>.metadata.json` written only if absent, `version-hint.text` for readers | Iceberg's file-system ("Hadoop") tables | Apache-2.0 | `lakebed_iceberg::commit` |
| Commits to tables without a catalog: `v<N>.metadata.json` written only if absent, `version-hint.text` for readers | Iceberg's file-system ("Hadoop") tables | Apache-2.0 | `lakekit_iceberg::commit` |
| Avro-safe names for partition fields (`_x` and the character's code in hex) | Iceberg's Java implementation | Apache-2.0 | Manifest writing |
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
# lakebed
# lakekit

Table formats read and written without I/O of their own, in Rust, for WebAssembly and native targets. Each format's
metadata (Apache Iceberg first) is parsed and written by plain code that asks a `Host` for bytes: the application
implements the host over its object store, so the same crate runs in a browser, a Cloudflare Worker, a CLI, or a
server.

lakebed is what [Timelake](https://github.com/polythings/timelake) and [lakeglass](https://github.com/polythings/lakeglass)
lakekit is what [Timelake](https://github.com/polythings/timelake) and [lakeglass](https://github.com/polythings/lakeglass)
share. The official Rust crates for these formats build on tokio and HTTP clients, which don't compile to
`wasm32-unknown-unknown`.

## Crates

| Crate | What it does |
|---|---|
| [`lakebed-core`](crates/lakebed-core) | The `Host` trait (ranged reads, listing, conditional writes, clock, randomness), errors, an in-memory host |
| [`lakebed-iceberg`](crates/lakebed-iceberg) | Apache Iceberg: metadata, manifests, and commits |
| [`lakebed`](crates/lakebed) | Re-exports the format crates, one feature each |
| [`lakekit-core`](crates/lakekit-core) | The `Host` trait (ranged reads, listing, conditional writes, clock, randomness), errors, an in-memory host |
| [`lakekit-iceberg`](crates/lakekit-iceberg) | Apache Iceberg: metadata, manifests, and commits |
| [`lakekit`](crates/lakekit) | Re-exports the format crates, one feature each |

More formats get their own crate as they are added (spec 2).

Expand All @@ -30,12 +30,12 @@ More formats get their own crate as they are added (spec 2).
| Schema as an Arrow schema with field ids | Feature `arrow` |
| A data file's Iceberg statistics from its Parquet footer | Feature `parquet-stats` |

What lakebed writes, iceberg-rust reads, and the other way round: the tests commit to one table with both in turn.
What lakekit writes, iceberg-rust reads, and the other way round: the tests commit to one table with both in turn.

```rust
use lakebed_iceberg::Table;
use lakekit_iceberg::Table;

async fn files<H: lakebed_core::Host>(host: &H) -> lakebed_core::Result<usize> {
async fn files<H: lakekit_core::Host>(host: &H) -> lakekit_core::Result<usize> {
let mut table = Table::load(host, "s3://bucket/warehouse/db/events").await?;
Ok(table.live_files(host).await?.len())
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "lakebed-core"
description = "The I/O boundary shared by lakebed's table format crates: a host trait for object stores, clocks, and randomness"
name = "lakekit-core"
description = "The I/O boundary shared by lakekit's table format crates: a host trait for object stores, clocks, and randomness"
readme = "README.md"
version.workspace = true
edition.workspace = true
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# lakebed-core
# lakekit-core

The I/O boundary of [lakebed](https://github.com/polythings/lakebed)'s table format crates. Format crates do no I/O of
The I/O boundary of [lakekit](https://github.com/polythings/lakekit)'s table format crates. Format crates do no I/O of
their own: they call a `Host` that the application implements over its object store, clock, and randomness, so they
build for `wasm32-unknown-unknown` and native targets alike.

Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
//! The I/O boundary shared by lakebed's table format crates. Format crates do no I/O of their own:
//! The I/O boundary shared by lakekit's table format crates. Format crates do no I/O of their own:
//! they read and write through a [`Host`] the application implements over its object store (an
//! `object_store` client natively, `fetch` or an R2 binding in WebAssembly), so they build for
//! `wasm32-unknown-unknown` and native targets alike.
//!
//! ```
//! use lakebed_core::{Host, MemoryHost};
//! use lakekit_core::{Host, MemoryHost};
//!
//! # block_on(async {
//! let host = MemoryHost::new(0);
//! host.put("memory://t/a", b"x".to_vec()).await?;
//! assert!(host.exists("memory://t/a").await?);
//! # Ok::<(), lakebed_core::Error>(())
//! # Ok::<(), lakekit_core::Error>(())
//! # }).unwrap();
//! # // MemoryHost's futures are ready at once, so one poll runs them.
//! # fn block_on<F: std::future::Future>(f: F) -> F::Output {
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "lakebed-iceberg"
name = "lakekit-iceberg"
description = "Apache Iceberg table metadata, manifests and commits without I/O of its own, for WebAssembly and native"
readme = "README.md"
version.workspace = true
Expand All @@ -20,7 +20,7 @@ arrow = ["dep:arrow-schema"]
parquet-stats = ["dep:parquet"]

[dependencies]
lakebed-core.workspace = true
lakekit-core.workspace = true
apache-avro.workspace = true
serde_json.workspace = true
futures-util.workspace = true
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# lakebed-iceberg
# lakekit-iceberg

[Apache Iceberg](https://iceberg.apache.org) tables read and written without I/O of their own, through a
[`lakebed_core::Host`](../lakebed-core), so they work in WebAssembly (browsers, Cloudflare Workers) as well as natively.
[`lakekit_core::Host`](../lakekit-core), so they work in WebAssembly (browsers, Cloudflare Workers) as well as natively.

| Feature | What it adds |
|---|---|
Expand All @@ -10,6 +10,6 @@
| `arrow` | The schema as an Arrow schema, with field ids |
| `parquet-stats` | A data file's Iceberg statistics (counts, bounds, split offsets) from its Parquet footer |

The tests commit to one table with lakebed and iceberg-rust in turn, each reading what the other wrote.
The tests commit to one table with lakekit and iceberg-rust in turn, each reading what the other wrote.

Apache License 2.0.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ use std::future::Future;
use std::hash::BuildHasher;
use std::sync::Arc;

use lakebed_core::{Host, uuid_v4};
use lakekit_core::{Host, uuid_v4};

use crate::manifest::{
DataFile, Entry, ManifestContext, ManifestFile, ManifestWrite, Status, manifest_file, write_manifest,
Expand Down Expand Up @@ -469,7 +469,7 @@ pub fn counters<S: BuildHasher>(change: &Change, base: &HashMap<String, String,

#[cfg(test)]
mod tests {
use lakebed_core::MemoryHost;
use lakekit_core::MemoryHost;
use serde_json::json;

use super::*;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
//! Apache Iceberg table metadata, manifests and commits, with no I/O of its own: everything is read
//! and written through a [`Host`](lakebed_core::Host) the application implements, so the crate
//! and written through a [`Host`](lakekit_core::Host) the application implements, so the crate
//! builds for `wasm32-unknown-unknown` (browsers, Cloudflare Workers) as well as native targets.
//!
//! - Reading (always): table metadata of format versions 1 to 3 ([`Metadata`]), schemas and
Expand All @@ -11,11 +11,11 @@
//! - `parquet-stats`: a data file's Iceberg statistics from its Parquet footer.
//!
//! ```
//! use lakebed_iceberg::{Datum, PrimitiveType};
//! use lakekit_iceberg::{Datum, PrimitiveType};
//!
//! let bound = Datum::from_bytes(&PrimitiveType::Long, &42i64.to_le_bytes())?;
//! assert_eq!(bound, Datum::Long(42));
//! # Ok::<(), lakebed_core::Error>(())
//! # Ok::<(), lakekit_core::Error>(())
//! ```

#[cfg(feature = "arrow")]
Expand All @@ -35,7 +35,7 @@ mod types;
#[cfg(feature = "write")]
pub use commit::{Change, CommitHooks, CommitInfo, CommitOptions, Extra, NoHooks, commit};
pub use datum::{Datum, from_be, minimal_be};
pub use lakebed_core::{Error, Result};
pub use lakekit_core::{Error, Result};
pub use manifest::{
Content, DataFile, Entry, FieldSummary, ManifestContent, ManifestFile, Status, read_manifest, read_manifest_list,
};
Expand Down Expand Up @@ -76,7 +76,7 @@ fn corrupt(msg: impl Into<String>) -> Error {
Error::Corrupt(msg.into())
}

/// Runs a future whose host calls complete at once ([`lakebed_core::MemoryHost`]'s do).
/// Runs a future whose host calls complete at once ([`lakekit_core::MemoryHost`]'s do).
#[cfg(test)]
fn block_on<F: std::future::Future>(f: F) -> F::Output {
let mut cx = std::task::Context::from_waker(std::task::Waker::noop());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
//! them, with `version-hint.text` naming the newest for stores that can't list. Tables a catalog
//! names with other file names load with [`Table::from_metadata_file`].

use lakebed_core::Host;
use lakekit_core::Host;

use crate::manifest::{Entry, ManifestFile, Status, read_manifest, read_manifest_list};
use crate::metadata::Metadata;
Expand Down Expand Up @@ -123,7 +123,7 @@ impl Table {

#[cfg(test)]
mod tests {
use lakebed_core::MemoryHost;
use lakekit_core::MemoryHost;

use super::*;
use crate::block_on;
Expand Down
Loading
Loading