diff --git a/README.md b/README.md index d9b6d9e..7968708 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,7 @@ An earlier version distilled runs into a shared library of verified skills with ``` packages/codemode-git/ the plugin: history connector, Artifacts sink, examples (Vectorize) examples/agent/ a stock Code Mode agent with a paginated `lamps` connector (a lava lamp wall) -benchmarks/dabstep/ DABstep harness: stock Code Mode vs Code Mode + codemode-git +benchmarks/dabstep/ DABstep harness: stock Code Mode vs Code Mode + codemode-git (see its README to run it) ``` ## Run the example diff --git a/benchmarks/dabstep/README.md b/benchmarks/dabstep/README.md new file mode 100644 index 0000000..bbace21 --- /dev/null +++ b/benchmarks/dabstep/README.md @@ -0,0 +1,143 @@ +# DABstep benchmark + +This harness runs Adyen's [DABstep](https://huggingface.co/datasets/adyen/DABstep) data-analysis benchmark through Code Mode agents on Cloudflare Workers, in two arms: + +| Arm | What the agent gets | +|---|---| +| `baseline` | Stock Code Mode: a read-only `data` connector over the DABstep context in D1. | +| `codemode-git-examples` | The same, plus codemode-git. Every run is a commit on the agent's own Artifacts repo, and the agent starts with the 3 closest answer-producing runs other agents committed, retrieved from Vectorize and shown as unverified examples. | + +Each task gets one fresh agent (a `Solver` Durable Object), with the same model, prompt and step budget in both arms. The default model is `@cf/zai-org/glm-5.3-flash` on Workers AI, with 20 steps. Results and analysis are in [docs/experiments.md](../../docs/experiments.md). The code for the reported full run is at tag [`dabstep-full1`](https://github.com/smithclay/codemode-git/tree/dabstep-full1). + +## Prerequisites + +- Node 22+ and Python 3. +- A Cloudflare account on Workers Paid, with Artifacts, Workers AI and Vectorize. +- The `cf` CLI, logged in (`npx cf login`). +- `npm install` at the repo root. The harness is a workspace and uses the local `@smithclay/codemode-git`. + +All commands below run from `benchmarks/dabstep`. + +## 1. Download the data + +```bash +scripts/fetch.sh +``` + +This fetches the context files (payments, fees, merchant data, the manual), the `dev` and `all` task splits, and Adyen's official scorer into `data/`, which is gitignored. The DABstep data is CC BY 4.0. Only the 10 `dev` tasks have public answers; the 450 in `all` are scored by the leaderboard. + +## 2. Load the context into D1 + +Build one SQL file from the context. Tables keep the original file and column names, so the manual's terms map onto them. + +```bash +python3 scripts/build_sql.py > data/load.sql +``` + +Create a database and put its id in `cloudflare.config.ts` (`DB: bindings.d1({ name: "dabstep", id: "…" })`). The id in the repo is the original author's database. + +```bash +npx cf d1 create dabstep +``` + +Then load the SQL (about 25 MB). With `cf`, apply it as a single migration: + +```bash +mkdir -p data/sql && cp data/load.sql data/sql/0001_dabstep.sql +npx cf d1 migrations apply --dir data/sql +``` + +> This step wasn't re-run for this README; the original database was loaded earlier. If the migration hits a size limit, use D1's bulk import instead: add the database to a minimal Wrangler config (`d1_databases`) and run `npx wrangler d1 execute dabstep --remote --file data/load.sql`. + +## 3. Create the other resources + +- **Artifacts namespace** (one repo per agent): + + ```bash + npx cf artifacts namespaces create --namespace codemode-git + ``` + +- **Vectorize index** `codemode-git-examples` (1024 dimensions, cosine), with metadata indexes on `scope`, `ok` and `task`. The commands are in the [root README](../../README.md#examples-learning-from-other-agents-runs-optional). The examples arm needs the metadata indexes; without them, filtered queries return nothing. + +## 4. Start the harness Worker + +**Locally,** with remote bindings for D1, Artifacts, Workers AI and Vectorize: + +```bash +npx cf dev --port 8790 +``` + +**Or deployed.** Every request to a deployed Worker needs `HARNESS_TOKEN` as a bearer token, and the Worker rejects everything without it. Set it at deploy time so it's never open: + +```bash +echo '{"HARNESS_TOKEN": ""}' > data/secrets.json +npx cf deploy --secrets-file data/secrets.json +``` + +Before spending on model calls, run the self-test. It checks the Code Mode runtime against real D1 and does an examples round trip through Vectorize, with no chat-model calls: + +```bash +curl -X POST localhost:8790/selftest +``` + +## 5. Run the tasks + +`scripts/run.mjs` sends tasks to `POST /solve`, one fresh agent per task, and appends results to `results/-.jsonl`, which is gitignored. It's resumable: finished tasks are skipped on a rerun. A dropped connection reconnects to the same agent, which rejoins its run. Other infrastructure failures retry the task with a fresh agent, up to 3 attempts, so failures don't bias results toward easy tasks. Pass `--url` and set `HARNESS_TOKEN` in the environment for a deployed Worker. + +Start with a smoke test on the graded dev tasks: + +```bash +node scripts/run.mjs --run smoke --arm baseline --split dev --limit 5 --url http://localhost:8790 +``` + +The full run (`full1`) used all 450 tasks, both arms side by side in two terminals, with rolling concurrency 8: + +```bash +node scripts/run.mjs --run full1 --arm baseline --split all --concurrency 8 --rolling --url +node scripts/run.mjs --run full1 --arm codemode-git-examples --split all --concurrency 8 --rolling --url +``` + +- **Time and cost:** each arm took about 8.5 hours of wall time. Workers AI spend was $26.76 for both arms, including retries. +- **Spend guard:** `--budget-tokens` (default 175M) stops launching new tasks once finished tasks have used that many tokens. +- **Task order matters for the examples arm,** since agents learn only from earlier runs. `data/tasks/all.jsonl` is used in its published order. +- **Without `--rolling`,** tasks go out in waves, and the examples arm waits `--settle` seconds (default 120) between waves. Vectorize metadata filters see new vectors only after about 1–2 minutes. +- **Other flags:** `--model`, `--max-steps` and `--tasks `. + +## 6. Score + +```bash +python3 scripts/score.py results/full1-baseline.jsonl results/full1-codemode-git-examples.jsonl +``` + +This scores the `dev` tasks with the official scorer, and writes `.submission.jsonl` (`{task_id, agent_answer}`) for the [DABstep leaderboard](https://huggingface.co/spaces/adyen/DABstep). Accuracy on all 450 tasks comes from submitting there. The leaderboard publishes per-task scores in the dataset under `data/task_scores/`. + +## 7. Analyze + +- **Paired comparison:** accuracy (with McNemar counts), step-limit exhaustion, code runs, tokens and time, plus a learning curve by position in the run: + + ```bash + python3 scripts/compare.py results/full1-baseline.jsonl results/full1-codemode-git-examples.jsonl --slice all + ``` + +- **Learning curve from the official per-task scores:** download your two `data/task_scores/` files from the dataset and save them as `Baseline.jsonl` and `with_Examples.jsonl` in one directory, then: + + ```bash + python3 scripts/learning_curve.py + ``` + +## Smaller experiments + +DABstep's 450 tasks come from about 113 question templates, so retrieval has related runs to find. For cheaper A/B runs: + +- `scripts/make_slice.py --name ab25 --sample 15` samples test-split tasks (ungraded material to learn from), followed by every dev task (graded). +- `scripts/make_template_slice.py` concentrates on the templates of chosen dev tasks, where learning across agents has the best chance to apply. + +Both write `data/tasks/.jsonl`. Run a slice with `--split `, and compare with `--slice `. The experiments log in [docs/experiments.md](../../docs/experiments.md) records the slices used (`ab25`, `tpl36`) and what they showed. + +## Tests + +```bash +npm test --workspace benchmarks/dabstep +``` + +These are unit tests for the agent loop's tool-call parsing and result clipping.