From 4cd0afd37e2b237e68c5b50e552c1f7dae4a2af2 Mon Sep 17 00:00:00 2001 From: Eddie A Tejeda <669988+eddietejeda@users.noreply.github.com> Date: Sun, 18 Oct 2026 08:50:35 -0700 Subject: [PATCH 1/3] README: simplify, focus on the plugin for developers --- README.md | 217 ++++++++++++++++++++++++------------------------------ 1 file changed, 95 insertions(+), 122 deletions(-) diff --git a/README.md b/README.md index 4355524..15c3240 100644 --- a/README.md +++ b/README.md @@ -1,147 +1,120 @@ # hotdata-tabular-memory -Makes [Hotdata](https://hotdata.dev) the default database when a Claude agent meets data -too big for its context. One database per project, found by every agent working in it. -Definitions every agent applies the same way. Results that stay on the server. - -**No dependencies.** Python's standard library and the `hotdata` CLI — the Rust SDK -already compiled and already holding your credentials. 224KB installed, nothing to -bootstrap. - -## Why - -In a 240-session benchmark, agents given a 2.96M-row table pushed computation to it and -pulled back a median of 88 characters per query — the largest result ever was 1,468 — -without anything telling them to. That is the product. What they did badly at swarm -scale was repeat work (a quarter of all queries, rising with every agent added), -re-orient (`SELECT * LIMIT 2` ran 102 times across 14 agents), and invent definitions -(every correctness failure in 268 sessions was a filter the question never specified). - -This plugin keeps the first behaviour and removes the other three. Measured against -two alternatives on Claude Opus 5, two agents each, twenty sessions, the same fifty -questions twice. The baseline is Claude with **no Hotdata at all** — the parquet on disk -and whatever the machine has, which turned out to be DuckDB. - -| | No Hotdata | Hotdata CLI | Plugin v2.1 | -| :- | :- | :- | :- | -| Correct | 95.0% | 97.0% | 97.5% | -| Right then wrong | 3 | 2 | 0 | -| Turns / session | 5.0 | 7.2 | 3.4 | -| Tokens / session | 138k | 191k | 97k | -| Tokens / correct answer | 29,125 | 39,414 | 19,981 | -| $ / correct answer | $0.053 | $0.061 | $0.045 | -| Seconds / session | 21s | 33s | 15s | - -Against working locally with no database: the same cost in dollars, **32% fewer turns, 30% fewer tokens and 29% less wall time**, with three accuracy points and three right-then-wrong flips recovered. Against the raw Hotdata CLI the gap is wider on every axis. - -Two things that result depends on, both measured: - -**The benchmark is 48MB on the agent's own machine** — the case most favourable to the -local baseline, where DuckDB reads the parquet directly with no network, auth or -transfer. Data larger than the machine, shared between agents, or already loaded is -outside what it tests. - -**The gain depends on the model batching.** Repeating the run on Haiku 5.5 cut turns by -2% and tokens by 13%, against 32% and 30% on Opus 5, and lost 1.5 points of accuracy. -Haiku 5.5 calls `hd batch` as often as Opus in absolute terms but makes 70% more calls -overall — it uses the affordance and keeps working the old way beside it. It also -answers 97.5% of these questions unaided, so there was no accuracy headroom to recover. -`bench/README.md` has the figures. - -**Agents sharing work is tested separately**, in `swarm/`: many agents whose tasks -overlap, where the shared work is classification an agent pays to redo. In the pilot, -agents sharing a store reused every label already recorded, but the plugin cost more -per task than agents sharing a folder, through extra round trips rather than the store. - -## What it does - -| Piece | What happens | -| :- | :- | -| `hd load ` | the file becomes a table in the project's database, created on first use | -| `hd query ""` | `{{name}}` expands to a definition; an unbounded scan is refused; rows over the cap stay on the server | -| `hd define ""` | a filter or derived set every agent in the project shares, versioned | -| `hd share ""` | a query result becomes a table without a row entering anyone's context | -| `hd fork` / `hd attach` | an isolated copy to experiment in; another database's tables made visible here | -| SessionStart hook | the database, its tables, a sample, the definitions and the dialect notes, from a cache, in ~40ms | -| PreToolUse guard | reading a data file over 8MB into context is refused, with the `hd load` command in the refusal | +A Claude Code plugin that gives agents a shared SQL database for data too big for +context. Each project gets one [Hotdata](https://hotdata.dev) database. Every agent in +the project finds it, uses the same named filters, and keeps large results on the server. -The guard is a refusal rather than advice because the benchmark found that refusals held -and instructions did not. +## Install -## The model +```sh +/plugin marketplace add hotdata-dev/claude-plugins +/plugin install hotdata-tabular-memory@hotdata +``` -**One database per project.** The project is named by its repository root, so every -agent in the repo resolves the same database; a `project` option pins the name. +You need a Hotdata account, logged in on the machine. The first `hd load` creates the +project's database. Nothing else to set up. -**Definitions, not restated filters.** A definition is a saved query on the server -(`POST /v1/queries`): named, versioned, hashed. In SQL, `{{clean_set}}` expands to it as -a subquery. The engine has no DDL and so no views; this is a view by expansion. An agent -writing `FROM {{clean_set}}` cannot apply a different filter. +## How it works -**Fork for isolation, attach for sharing, share for reuse.** All three are existing -Hotdata primitives; the plugin removes the ceremony and never relies on the CLI's -active-database pointer, which is global state two agents would fight over. +| Piece | What it does | +| :- | :- | +| `hd` command | Loads files, runs SQL, manages shared definitions. On `PATH` while the plugin is enabled. | +| SessionStart hook | Tells the agent the database, its tables, a sample, and the definitions. Served from a cache in about 40ms. | +| PreToolUse guard | Refuses to read a data file over 8MB into context, and suggests `hd load` instead. | +| `tabular-memory` skill | Tells Claude when and how to use `hd`. | -## Install +**One database per project.** The repository root names the project, so every agent in +the repo resolves the same database. -```sh -claude plugin marketplace add /path/to/hotdata-tabular-memory -claude plugin install hotdata-tabular-memory@hotdata -``` +**Definitions instead of repeated filters.** A definition is a named, versioned query +stored on the server. In SQL, `{{name}}` expands to it as a subquery. Two agents that +both write `FROM {{clean_trips}}` cannot apply different filters. + +**Results stay on the server.** `hd query` stops at a row cap. `hd share` saves a result +as a table, so other agents can read it without the rows entering anyone's context. -Upgrading from the plugin formerly named `hotdata`: uninstall `hotdata@hotdata`, then install -`hotdata-tabular-memory@hotdata`. Version 3.0.0 is the rename; the behaviour is 2.1.3's. +## Commands -That is enough: the first `hd load` creates the project's database. To point a project -at a database that already exists, `--config database=`. +```sh +hd load data/trips.parquet # file or URL becomes a table +hd query "SELECT count(*) FROM {{clean}}" # run SQL +hd batch "a: SELECT ..." "b: SELECT ..." # several queries in one call +hd define clean "SELECT * FROM trips WHERE fare > 0" +hd defs # list definitions +hd share top_zones "SELECT ..." # save a result as a table +hd fork # isolated copy to experiment in +hd attach # read another database's tables +hd status # project, database, tables, definitions +hd orient # what an agent sees at session start +``` -To stop Claude asking before every read, add the read-only commands to -`permissions.allow` in `~/.claude/settings.json`; leave `load`, `define`, `share`, -`fork` and `attach` out so a change still asks: +To skip permission prompts for read-only commands, add this to `permissions.allow` in +`~/.claude/settings.json`. Commands that change data still ask. ```json -"allow": ["Bash(hd query:*)", "Bash(hd defs)", "Bash(hd status)", "Bash(hd orient:*)"] +"allow": ["Bash(hd query:*)", "Bash(hd batch:*)", "Bash(hd defs)", "Bash(hd status)", "Bash(hd orient:*)"] ``` -A plugin with a top-level `bin/` is not installable on claude.ai or Cowork; call -`scripts/hd.py` directly there. - ## Configuration -| Option | Default | | +Set options with `--config