lakekit 0.1: lakekit-core and lakekit-iceberg

.github/dependabot.yml

@@ -0,0 +1,24 @@
+# Dependabot keeps dependencies current: one pull request a week per ecosystem with minor and patch updates, and one
+# per major update. A new version is proposed once it's 7 days old (security updates at once).
+version: 2
+
+updates:
+  - package-ecosystem: github-actions
+    directory: /
+    schedule: { interval: weekly }
+    cooldown: { default-days: 7 }
+    groups:
+      actions: { update-types: [minor, patch] }
+
+  - package-ecosystem: cargo
+    directory: /
+    schedule: { interval: weekly }
+    cooldown: { default-days: 7 }
+    groups:
+      cargo: { update-types: [minor, patch] }
+    ignore:
+      # iceberg (pinned with =, a test dependency) fixes the Arrow and Parquet majors: move them with it, by hand.
+      - dependency-name: "arrow-*"
+        update-types: [version-update:semver-major]
+      - dependency-name: parquet
+        update-types: [version-update:semver-major]

.github/workflows/ci.yml

@@ -0,0 +1,54 @@
+# Checks every push to main and every pull request. Needs no secrets.
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.event_name == 'pull_request' && github.ref || github.sha }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_INCREMENTAL: 0
+
+jobs:
+  native:
+    name: Native (format, lint, tests)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - run: rustup toolchain install   # from rust-toolchain.toml
+      - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
+      - run: cargo fmt --all --check
+      - run: cargo clippy --workspace --all-features --all-targets -- -D warnings
+      - run: cargo test --workspace --all-features
+      # Each feature builds on its own, and with none, without warnings.
+      - name: Features one at a time
+        env:
+          RUSTFLAGS: -D warnings
+        run: |
+          cargo check -p lakebed-iceberg --no-default-features
+          cargo check -p lakebed-iceberg --features write
+          cargo check -p lakebed-iceberg --features arrow
+          cargo check -p lakebed-iceberg --features parquet-stats
+
+  wasm:
+    name: WebAssembly (browser and WASI targets)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - run: rustup toolchain install   # from rust-toolchain.toml
+      - run: rustup target add wasm32-unknown-unknown wasm32-wasip2
+      - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3
+      - run: cargo clippy --workspace --all-features --target wasm32-unknown-unknown -- -D warnings
+      - run: cargo build --workspace --all-features --target wasm32-wasip2

.gitignore

@@ -0,0 +1,3 @@
+/target
+.env*
+.DS_Store

.pre-commit-config.yaml

@@ -0,0 +1,38 @@
+# Git hooks, run by prek (https://prek.j178.dev): `uv tool install prek && prek install --install-hooks`.
+# On commit: formatters and fast checks on the staged files. On push: clippy, natively and for wasm32.
+# Everything: `prek run --all-files --hook-stage pre-push`. CI runs the same checks plus the tests.
+default_install_hook_types: [pre-commit, pre-push]
+
+repos:
+  - repo: builtin
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-merge-conflict
+      - id: check-yaml
+      - id: check-toml
+      - id: check-added-large-files
+        args: [--maxkb=1024]
+      - id: detect-private-key
+
+  - repo: local
+    hooks:
+      - id: rustfmt
+        name: rustfmt
+        entry: rustfmt --edition 2024
+        language: system
+        types: [rust]
+      - id: clippy
+        name: clippy
+        entry: cargo clippy --workspace --all-features --all-targets --quiet -- -D warnings
+        language: system
+        types: [rust]
+        pass_filenames: false
+        stages: [pre-push]
+      - id: clippy-wasm
+        name: clippy (wasm32)
+        entry: cargo clippy --workspace --all-features --target wasm32-unknown-unknown --quiet -- -D warnings
+        language: system
+        types: [rust]
+        pass_filenames: false
+        stages: [pre-push]

AGENTS.md

@@ -0,0 +1,30 @@
+# Agent instructions
+
+lakebed: table formats (Apache Iceberg first) read and written in Rust without I/O of their own, for WebAssembly and
+native targets. One crate per format (`crates/lakebed-<format>`) on a shared I/O boundary (`crates/lakebed-core`).
+Timelake and lakeglass use it.
+
+## Before changing code
+
+- Follow [CONTRIBUTING.md](CONTRIBUTING.md). It is the code guide, and it applies to you.
+- For anything beyond a small fix, check the specs ([spec/README.md](spec/README.md)). Agree on the design with the user
+  before building something new, and record the decision in the spec's Decisions table.
+
+## Rules
+
+- **Done means checked.** Before you report work as finished, run `prek run --all-files --hook-stage pre-push` and
+  `cargo test --workspace --all-features`. Report failures as failures.
+- **Fix the code, not the checks.** Don't loosen lint or format configs. Don't add `#[allow]`. Use
+  `#[expect(..., reason = "...")]` only when the tool is genuinely wrong.
+- **Keep things in sync:** a change in behaviour updates its spec (status, design, history), the tests, and
+  `CHANGELOG.md` (under `Unreleased`) in the same commit.
+- **Everything builds for WASM.** Every crate builds for `wasm32-unknown-unknown`: no threads, no blocking I/O, no
+  `std::time`, no C dependencies. All I/O goes through the `Host` trait. CI checks it.
+- **Readers don't trust files.** Bytes from a store can be anything: check lengths, return errors, never panic.
+- **Interop is tested.** What a format crate writes must read in that format's reference implementation and the other
+  way round; the tests show it.
+- **Every behaviour a caller might want to change is an option**, named for its mechanism.
+- **Ideas, not code.** Never copy or translate another project's code. Credit ideas in `PRIOR_ART.md`
+  (CONTRIBUTING.md, "Credit").
+- **Tests run offline.** No network, cloud accounts, or real buckets in tests.
+- **Never commit secrets** or `.env*` files.

CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to lakebed. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
+versions follow [Semantic Versioning](https://semver.org/).
+
+## Unreleased
+
+### Added
+
+- `lakebed-core`: the `Host` trait for object stores, clocks, and randomness; `MemoryHost`; one `Error` type.
+- `lakebed-iceberg`: Iceberg table metadata, schemas, partition specs, manifest lists, and manifests read for format
+  versions 1 to 3, including delete files and deletion vector references; values in Iceberg's single-value encoding.
+- `lakebed-iceberg` feature `write`: format v2 manifests, manifest lists, and metadata updates, and commits to
+  file-system tables with options (`CommitOptions`) and hooks (`CommitHooks`). A writer whose clock is behind the
+  table's last update dates its commit at that update, so the snapshot log stays in order.
+- `lakebed-iceberg` features `arrow` (schemas as Arrow schemas with field ids) and `parquet-stats` (a data file's
+  Iceberg statistics from its Parquet footer; a bound or null count only when every row group gives it exactly, and
+  decimals as Iceberg's shortest big-endian bytes).
+- `lakebed`: re-exports the format crates behind a feature each.
+
+The Iceberg code started as Timelake's shared writer (`timelake-format`), made general.

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

CONTRIBUTING.md

@@ -0,0 +1,85 @@
+# Contributing to lakebed
+
+This is the code guide for everyone who changes this repository, people and AI agents alike. Formatters and linters
+enforce the mechanical rules (see [Checks](#checks)). This guide covers what they can't: how code should read.
+[spec/](spec/README.md) holds the design.
+
+## Principles
+
+1. **Readable first.** Someone new should understand a function from its name, its signature, and one pass through
+   the body.
+2. **One crate per format.** A format's crate depends on `lakebed-core` and on nothing of another format's, so adding
+   one never touches the others. Formats are shown as they are: no shared model flattens their differences.
+3. **No I/O of our own.** Crates read and write through `lakebed_core::Host`. They never open sockets, files, or
+   threads, so they build for `wasm32-unknown-unknown`.
+4. **One definition of each thing.** Before writing a helper, constant, or type, look for an existing one.
+5. **Behaviour, spec, and tests change together.** A change users can see updates its spec, adds or updates tests, and
+   adds a line to [CHANGELOG.md](CHANGELOG.md).
+6. **Behaviour is configurable.** Where callers could reasonably want something else (retries, limits, what to write),
+   it is an option with a default, named for its mechanism (`merge_manifests_at`, not `fast`).
+
+## Comments and docs
+
+- **Public items get a short doc comment**: what it does, and anything a caller must know (errors, units,
+  invariants). One to three lines is usually enough.
+- **Comments explain why, not what.** Explain a non-obvious choice, a workaround, or a rule from a format's spec, with
+  a link to the spec section.
+- **No long comments.** If an explanation needs a page, it belongs in a spec. Link to it from the code.
+
+## Credit
+
+lakebed learns from other projects' ideas, never from copying their code. [PRIOR_ART.md](PRIOR_ART.md) lists the
+ideas we took.
+
+- **Never copy code**, whatever its license: not by pasting, and not by translating it into another language.
+- **Credit ideas from open-source projects** with a row in PRIOR_ART.md and a short comment where the idea is
+  implemented.
+- **Don't name open-core, dual-licensed, or source-available projects, or commercial products, as a source.**
+
+## Rust
+
+- **Errors:** `lakebed_core::Error`, with the variant that says what a caller can do about it (`NotFound`, `Conflict`,
+  `CommitUnknown`, `Unsupported`, `Corrupt`). No `unwrap()` outside tests.
+- **Types over strings:** enums and newtypes instead of strings and magic numbers. Parse once at the boundary.
+- **Don't trust files.** Check lengths and offsets before using them, cap allocations, and return an error instead of
+  panicking on a malformed file. Tests feed readers truncated and garbage input.
+- **Async:** host futures aren't `Send` (JavaScript promises aren't). Never block inside async code.
+- **Suppressing a lint:** `#[expect(lint, reason = "...")]` with a real reason. Never `#[allow]`.
+
+## Tests
+
+- **A bug fix starts with a test that fails**, then the fix.
+- **Tests run offline**, against `MemoryHost` or a host over the reference implementation's in-memory storage.
+- **Interop:** writers are tested against the format's reference reader, and readers against the reference writer.
+
+## Checks
+
+Install once: `uv tool install prek && prek install --install-hooks`. Hooks then run on commit and on push. Before you
+push, or before calling work done:
+
+```bash
+prek run --all-files --hook-stage pre-push   # rustfmt, clippy (native and wasm32)
+cargo test --workspace --all-features        # unit, doc, and interop tests
+```
+
+## Dependencies
+
+Keep dependencies on their newest versions; check the latest release when adding one. Arrow and Parquet follow the
+major version iceberg-rust's newest release uses (the interop tests build both). Actions are pinned to a commit with
+the version in a comment (`uses: actions/checkout@<sha> # v7.0.1`).
+
+## Commits
+
+- **Subject:** a short imperative sentence with the area first, for example `Core: …`, `Iceberg: …`, or `Spec: …`.
+  Add a body when the why isn't obvious.
+- **Scope:** one logical change per commit.
+
+## Changelog
+
+[CHANGELOG.md](CHANGELOG.md) is for users, not a list of commits. Anything a user can notice gets one line under
+`## Unreleased`, in `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, or `Security`, in the same commit as the
+change.
+
+## License
+
+Contributions are licensed under the Apache License 2.0, like the rest of the project (see [LICENSE](LICENSE)).