Spec: a common table view and data file formats

.github/workflows/ci.yml

@@ -39,6 +39,10 @@ jobs:
           cargo check -p lakekit-iceberg --features write
           cargo check -p lakekit-iceberg --features arrow
           cargo check -p lakekit-iceberg --features parquet-stats
+          cargo check -p lakekit-core --no-default-features
+          cargo check -p lakekit-core --features tokio
+          cargo check -p lakekit-core --features getrandom
+          cargo check -p lakekit --features object-store
 
   wasm:
     name: WebAssembly (browser and WASI targets)

AGENTS.md

@@ -19,7 +19,7 @@ Timelake and lakeglass use it.
 - **Keep things in sync:** a change in behaviour updates its spec (status, design, history), the tests, and
   `CHANGELOG.md` (under `Unreleased`) in the same commit.
 - **Everything builds for WASM.** Every crate builds for `wasm32-unknown-unknown`: no threads, no blocking I/O, no
-  `std::time`, no C dependencies. All I/O goes through the `Host` trait. CI checks it.
+  `std::time`, no C dependencies. All I/O goes through the `Store`, `Clock`, and `Random` traits. CI checks it.
 - **Readers don't trust files.** Bytes from a store can be anything: check lengths, return errors, never panic.
 - **Interop is tested.** What a format crate writes must read in that format's reference implementation and the other
   way round; the tests show it.

CHANGELOG.md

@@ -7,7 +7,11 @@ versions follow [Semantic Versioning](https://semver.org/).
 
 ### Added
 
-- `lakekit-core`: the `Host` trait for object stores, clocks, and randomness; `MemoryHost`; one `Error` type.
+- `lakekit-core`: loko's `Store`, `Clock`, and `Random` services with their test and native implementations
+  (`TokioClock` with feature `tokio`, `OsRandom` with feature `getrandom`); `Services`, the three borrowed
+  together for commits; one `Error` type, which a store's error converts into.
+- `lakekit` feature `object-store`: a `Store` over an `object_store` client (S3, GCS, Azure, local files, memory),
+  from a client or a URL (`loko-object-store`).
 - `lakekit-iceberg`: Iceberg table metadata, schemas, partition specs, manifest lists, and manifests read for format
   versions 1 to 3, including delete files and deletion vector references; values in Iceberg's single-value encoding.
 - `lakekit-iceberg` feature `write`: format v2 manifests, manifest lists, and metadata updates, and commits to
@@ -23,5 +27,10 @@ versions follow [Semantic Versioning](https://semver.org/).
 - The project is named lakekit, and its crates `lakekit-core`, `lakekit-iceberg`, and `lakekit`; it was lakebed
   (`lakebed-core`, …) until 2026-10-21. Nothing was published under the old names; GitHub redirects the old
   repository URL.
+- The I/O boundary is three traits, `Store` (objects), `Clock` (the time and a timer), and `Random` (random numbers),
+  instead of one `Host`; `MemoryHost` is `MemoryServices`. UUIDs come from the `uuid` crate.
+- The three traits are [loko](https://github.com/polythings/loko)'s, and `Clock::sleep` takes a `Duration`. The
+  `object_store` adapter is `loko-object-store`, re-exported as `lakekit::object_store`; `lakekit-object-store` is
+  gone (it was never published).
 
 The Iceberg code started as Timelake's shared writer (`timelake-format`), made general.

CONTRIBUTING.md

@@ -10,7 +10,8 @@ enforce the mechanical rules (see [Checks](#checks)). This guide covers what the
    the body.
 2. **One crate per format.** A format's crate depends on `lakekit-core` and on nothing of another format's, so adding
    one never touches the others. Formats are shown as they are: no shared model flattens their differences.
-3. **No I/O of our own.** Crates read and write through `lakekit_core::Host`. They never open sockets, files, or
+3. **No I/O of our own.** Crates read and write through `lakekit_core::Store` and take the time and random numbers
+   from `Clock` and `Random`. They never open sockets, files, or
    threads, so they build for `wasm32-unknown-unknown`.
 4. **One definition of each thing.** Before writing a helper, constant, or type, look for an existing one.
 5. **Behaviour, spec, and tests change together.** A change users can see updates its spec, adds or updates tests, and
@@ -43,13 +44,13 @@ ideas we took.
 - **Types over strings:** enums and newtypes instead of strings and magic numbers. Parse once at the boundary.
 - **Don't trust files.** Check lengths and offsets before using them, cap allocations, and return an error instead of
   panicking on a malformed file. Tests feed readers truncated and garbage input.
-- **Async:** host futures aren't `Send` (JavaScript promises aren't). Never block inside async code.
+- **Async:** store and clock futures aren't `Send` (JavaScript promises aren't). Never block inside async code.
 - **Suppressing a lint:** `#[expect(lint, reason = "...")]` with a real reason. Never `#[allow]`.
 
 ## Tests
 
 - **A bug fix starts with a test that fails**, then the fix.
-- **Tests run offline**, against `MemoryHost` or a host over the reference implementation's in-memory storage.
+- **Tests run offline**, against a `MemoryStore` with a `TestClock` and a `SeededRandom`, or a store over the reference implementation's in-memory storage.
 - **Interop:** writers are tested against the format's reference reader, and readers against the reference writer.
 
 ## Checks

Cargo.toml

@@ -14,6 +14,14 @@ categories = ["database-implementations", "wasm"]
 [workspace.dependencies]
 lakekit-core = { path = "crates/lakekit-core", version = "0.1.0" }
 lakekit-iceberg = { path = "crates/lakekit-iceberg", version = "0.1.0", default-features = false }
+# loko provides the Store, Clock, and Random services. From git until loko 0.2.0 is on crates.io
+# (polythings/loko#1); publishing uses the version.
+loko-store = { git = "https://github.com/polythings/loko", rev = "6c79750109b7929e927a582e92570105da625714", version = "0.2.0" }
+loko-clock = { git = "https://github.com/polythings/loko", rev = "6c79750109b7929e927a582e92570105da625714", version = "0.2.0" }
+loko-random = { git = "https://github.com/polythings/loko", rev = "6c79750109b7929e927a582e92570105da625714", version = "0.2.0" }
+loko-object-store = { git = "https://github.com/polythings/loko", rev = "6c79750109b7929e927a582e92570105da625714", version = "0.2.0" }
+object_store = { version = "0.14", default-features = false }
+url = "2"
 thiserror = "2"
 serde_json = "1"
 apache-avro = { version = "0.22", default-features = false }

PRIOR_ART.md

@@ -11,12 +11,16 @@ never code: every line of lakekit is our own. How we credit is in [CONTRIBUTING.
 | [iceberg-rust](https://github.com/apache/iceberg-rust) | Apache-2.0 | The reference the interop tests check against (a test dependency only) |
 | [Apache Avro](https://avro.apache.org) (`apache-avro`) | Apache-2.0 | Reading and writing manifests and manifest lists |
 | [arrow-rs](https://github.com/apache/arrow-rs) (`arrow-schema`, `parquet`) | Apache-2.0 | Arrow schemas; Parquet footers for statistics |
+| [loko](https://github.com/polythings/loko) | Apache-2.0 | The `Store`, `Clock`, and `Random` services, and the `object_store` adapter |
+| [object_store](https://github.com/apache/arrow-rs-object-store) | Apache-2.0 / MIT | The stores `loko-object-store` adapts (re-exported by `lakekit`) |
+| [uuid](https://github.com/uuid-rs/uuid) | Apache-2.0 / MIT | UUIDs for file names, from a `Random` and a `Clock` |
 
 ## Ideas
 
 | Idea | From | License of the source | Where in lakekit |
 |---|---|---|---|
-| A format library with no I/O of its own: the application supplies storage through a trait the library calls | [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs) (its `Engine` trait) | Apache-2.0 | `lakekit-core`'s `Host` |
+| A format library with no I/O of its own: the application supplies storage through a trait the library calls | [delta-kernel-rs](https://github.com/delta-io/delta-kernel-rs) (its `Engine` trait) | Apache-2.0 | `lakekit-core`'s `Store`, `Clock`, and `Random` |
 | One crate per backend under a small shared core, re-exported by an umbrella crate with a feature each | [arrow-rs](https://github.com/apache/arrow-rs), [Apache OpenDAL](https://github.com/apache/opendal) | Apache-2.0 | The workspace layout |
 | Commits to tables without a catalog: `v<N>.metadata.json` written only if absent, `version-hint.text` for readers | Iceberg's file-system ("Hadoop") tables | Apache-2.0 | `lakekit_iceberg::commit` |
+| The time and randomness as separate, swappable services rather than ambient globals | [ZIO](https://github.com/zio/zio) (`Clock`, `Random`) | Apache-2.0 | `lakekit-core`'s `Clock` and `Random` |
 | Avro-safe names for partition fields (`_x` and the character's code in hex) | Iceberg's Java implementation | Apache-2.0 | Manifest writing |

README.md

@@ -1,9 +1,10 @@
 # lakekit
 
 Table formats read and written without I/O of their own, in Rust, for WebAssembly and native targets. Each format's
-metadata (Apache Iceberg first) is parsed and written by plain code that asks a `Host` for bytes: the application
-implements the host over its object store, so the same crate runs in a browser, a Cloudflare Worker, a CLI, or a
-server.
+metadata (Apache Iceberg first) is parsed and written by plain code that asks a `Store` for bytes, a `Clock` for the
+time, and a `Random` for random numbers ([loko](https://github.com/polythings/loko)'s services): the application
+supplies them, so the same crate runs in a browser, a
+Cloudflare Worker, a CLI, or a server.
 
 lakekit is what [Timelake](https://github.com/polythings/timelake) and [lakeglass](https://github.com/polythings/lakeglass)
 share. The official Rust crates for these formats build on tokio and HTTP clients, which don't compile to
@@ -13,11 +14,11 @@ share. The official Rust crates for these formats build on tokio and HTTP client
 
 | Crate | What it does |
 |---|---|
-| [`lakekit-core`](crates/lakekit-core) | The `Host` trait (ranged reads, listing, conditional writes, clock, randomness), errors, an in-memory host |
+| [`lakekit-core`](crates/lakekit-core) | loko's `Store`, `Clock`, and `Random` with their test and native implementations, `Services`, and one error type |
 | [`lakekit-iceberg`](crates/lakekit-iceberg) | Apache Iceberg: metadata, manifests, and commits |
-| [`lakekit`](crates/lakekit) | Re-exports the format crates, one feature each |
+| [`lakekit`](crates/lakekit) | Re-exports the crates, one feature each, and `loko-object-store` (a `Store` over an `object_store` client) |
 
-More formats get their own crate as they are added (spec 2).
+More formats get their own crate as they are added (spec 2). Store adapters are loko's (spec 4).
 
 ## Iceberg
 
@@ -33,14 +34,21 @@ More formats get their own crate as they are added (spec 2).
 What lakekit writes, iceberg-rust reads, and the other way round: the tests commit to one table with both in turn.
 
 ```rust
+use std::sync::Arc;
+
 use lakekit_iceberg::Table;
+use loko_object_store::ObjectStoreAdapter;
 
-async fn files<H: lakekit_core::Host>(host: &H) -> lakekit_core::Result<usize> {
-    let mut table = Table::load(host, "s3://bucket/warehouse/db/events").await?;
-    Ok(table.live_files(host).await?.len())
+async fn files(s3: Arc<dyn object_store::ObjectStore>) -> lakekit_core::Result<usize> {
+    let store = ObjectStoreAdapter::new(s3, "s3://bucket");
+    let mut table = Table::load(&store, "s3://bucket/warehouse/db/events").await?;
+    Ok(table.live_files(&store).await?.len())
 }
 ```
 
+Commits also need a `Clock` and a `Random`: `Services::new(&store, &TokioClock, &OsRandom)` natively, the application's
+own in WebAssembly.
+
 ## Development
 
 ```bash

crates/lakekit-core/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lakekit-core"
-description = "The I/O boundary shared by lakekit's table format crates: a host trait for object stores, clocks, and randomness"
+description = "The I/O boundary shared by lakekit's table format crates: loko's Store, Clock, and Random, and one error type"
 readme = "README.md"
 version.workspace = true
 edition.workspace = true
@@ -10,8 +10,22 @@ repository.workspace = true
 keywords.workspace = true
 categories.workspace = true
 
+[features]
+default = []
+# `TokioClock`: the system clock and tokio's timer. Native targets only.
+tokio = ["loko-clock/tokio"]
+# `OsRandom`: the operating system's random number generator. Native targets and WASI.
+getrandom = ["loko-random/getrandom"]
+
 [dependencies]
+loko-store.workspace = true
+loko-clock.workspace = true
+loko-random.workspace = true
 thiserror.workspace = true
+uuid = { version = "1", default-features = false }
+
+[package.metadata.docs.rs]
+all-features = true
 
 [lints]
 workspace = true

crates/lakekit-core/README.md

@@ -1,12 +1,20 @@
 # lakekit-core
 
 The I/O boundary of [lakekit](https://github.com/polythings/lakekit)'s table format crates. Format crates do no I/O of
-their own: they call a `Host` that the application implements over its object store, clock, and randomness, so they
-build for `wasm32-unknown-unknown` and native targets alike.
+their own: they read and write through a `Store`, take the time from a `Clock`, and take random numbers from a
+`Random`, all supplied by the application ([loko](https://github.com/polythings/loko)'s services, re-exported here), so they build for `wasm32-unknown-unknown` and native targets alike.
 
-- `Host`: get, ranged get, exists, put, put-if-absent (the commit primitive), multipart upload, list, delete, the time,
-  random numbers, and waiting. Its futures aren't `Send`, so a host can wrap JavaScript promises.
-- `MemoryHost`: a host in memory with a fixed clock and seeded randomness, for tests.
-- `Error`: one error type, whose variant says what a caller can do (`NotFound`, `Conflict`, `CommitUnknown`, …).
+- `Store`: get, ranged get, exists, put, put-if-absent (the commit primitive), multipart upload, list, delete. Its
+  futures aren't `Send`, so a store can wrap JavaScript promises. `loko-object-store` implements it over
+  `object_store`.
+- `Clock`: the time and a timer. `TokioClock` (feature `tokio`, native only) uses the system clock and tokio's timer;
+  `TestClock` moves only when told.
+- `Random`: random numbers. `OsRandom` (feature `getrandom`, native and WASI) uses the operating system's generator;
+  `SeededRandom` repeats a sequence from a seed.
+- `Services`: a store, a clock, and a random source, each borrowed, for APIs that need all three (commits):
+  `Services::new(&store, &clock, &random)`.
+- `uuid_v4`, `uuid_v7`: identifiers from a `Random` (and a `Clock`), with the `uuid` crate.
+- `Error`: one error type, whose variant says what a caller can do (`NotFound`, `Conflict`, `CommitUnknown`, …); a
+  store's error converts into it.
 
 Apache License 2.0.

crates/lakekit-core/src/error.rs

@@ -28,3 +28,31 @@ pub enum Error {
 
 /// `Result` with [`Error`].
 pub type Result<T, E = Error> = std::result::Result<T, E>;
+
+impl From<loko_store::Error> for Error {
+    fn from(e: loko_store::Error) -> Self {
+        match e {
+            loko_store::Error::NotFound(m) => Error::NotFound(m),
+            loko_store::Error::Invalid(m) => Error::Invalid(m),
+            loko_store::Error::Unsupported(m) => Error::Unsupported(m),
+            other => Error::Store(match other {
+                loko_store::Error::Store(m) => m,
+                other => other.to_string(),
+            }),
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn store_errors_keep_their_kind() {
+        let from = |e| Error::from(e).to_string();
+        assert_eq!(from(loko_store::Error::NotFound("a".into())), "not found: a");
+        assert_eq!(from(loko_store::Error::Invalid("b".into())), "invalid: b");
+        assert_eq!(from(loko_store::Error::Unsupported("c".into())), "unsupported: c");
+        assert_eq!(from(loko_store::Error::Store("d".into())), "store: d");
+    }
+}