Skip to content

feat(sample): arcli sample {list,show,load} for real datasets - #26

Merged
xe-nvdk merged 2 commits into
mainfrom
feat/sample-load
Sep 9, 2026
Merged

xe-nvdk merged 2 commits into
mainfrom
feat/sample-load

Conversation

@xe-nvdk

@xe-nvdk xe-nvdk commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds arcli sample list | show | load so a new user can put real data into their own Arc instead of writing one synthetic point and wondering what Arc is for.
  • sample load citibike downloads a month of NYC Citi Bike trips (31 files, 39.4 MiB, 1,003,770 rows), verifies every file against the published SHA-256, imports it, and prints two queries worth running against what just landed.
  • sample show citibike -o json prints every file URL and checksum, so anyone who would rather not fetch from us can download the data themselves and use arcli import parquet.
  • Datasets come from samples.basekick.net, the only Basekick-controlled host arcli contacts. Requests carry the same User-Agent and Arcli-Installation-Id as every other arcli request (the existing DO_NOT_TRACK / send_installation_id opt-outs apply) and never the token, the Arc endpoint, or anything about the user's data.
  • The README previously claimed "arcli never contacts Basekick or any third party." That is no longer true; the Privacy section now says exactly what is sent, including that fetching a file reveals your IP to whoever serves it.

Behaviour worth knowing

  • Creates the target database if absent; asks before adding to a measurement that already holds rows, so a second run cannot silently double the counts the printed queries report (--yes skips it, and a non-TTY refuses rather than hanging).
  • Interrupted downloads resume: a file already present with a matching checksum is not re-fetched; a mismatch is deleted and re-downloaded once before failing.
  • --download-only needs no Arc connection at all.
  • Import is admin-only server-side, so the preflight runs before any bytes are downloaded.

Review findings fixed before this PR

The internal deep review found two things that mattered, both fixed here:

  • Fetch had no fail-fast. An unrecognised build 403s on every part, so a user would have waited through 31 failures. A failure now cancels the rest: 4 requests instead of 31. Regression-tested.
  • Progress could print 31/31 downloaded on a run that then failed, because the counter incremented on errors too. It now counts only successes.

Also fixed: a goroutine leak on early return (stragglers could write into a directory the command had already deleted), manifest validation for empty part keys / negative sizes / colliding basenames, three independent timeouts converged onto --timeout, and a generic query fallback so a future dataset does not silently ship without the payoff.

One process note: my first fail-fast test passed while the code was broken and failed once it was fixed — it was measuring its own sleep, not the cancellation. Caught only by running it. Both regression tests are in the permanent suite.

Test plan

  • gofmt -l ., go vet ./..., go test -race -count=1 ./... — all clean
  • Live arcli sample load citibike against a local arc serve: 31/31 files, checksums verified, 1,003,770 rows imported in ~3s
  • Both printed queries pasted verbatim into arcli query — correct results (top station W 21 St & 6 Ave, 4,070; commute curve peaks 08:00 / 17:00)
  • Second run detects existing rows, prompts, and refuses cleanly in a non-TTY
  • --download-only with no Arc configured; --download-dir re-run reports 31 cached, zero re-downloads
  • Configuration matrix: no config file, --token without --endpoint, --endpoint only (auth disabled), ARC_ENDPOINT, -c missing connection, -o arrow, DO_NOT_TRACK=1
  • Installation id verified on the wire for both the manifest fetch and part downloads; Authorization never sent to the sample host
  • Every flag documented in the README verified to exist on the binary

Ignacio Van Droogenbroeck added 2 commits September 8, 2026 18:41
Load a real public dataset into your own Arc instead of writing one
synthetic point and wondering what Arc is for. `arcli sample load
citibike` downloads a month of NYC Citi Bike trips, verifies every file
against the published SHA-256, imports it, and prints two queries worth
running against what just landed.

- `sample list` — datasets and months, one row per loadable pair
- `sample show` — details, licence, time window; `-o json` prints every
  file URL and checksum so the data can be fetched without arcli and
  imported with `arcli import parquet`
- `sample load` — download, verify, import; creates the target database,
  resumes an interrupted run, and asks before adding to a measurement
  that already holds rows

Downloads are fetched from samples.basekick.net, the only Basekick-
controlled host arcli contacts. They carry the same User-Agent and
Arcli-Installation-Id as every other arcli request, honouring the
existing DO_NOT_TRACK and send_installation_id opt-outs, and never the
token, the Arc endpoint, or anything about the user's data. The README
privacy section said arcli never contacts Basekick; that is no longer
true and now says what is actually sent.

Fetching fails fast: a failure cancels the remaining downloads, so an
unrecognised build that 403s on every part costs 4 requests rather than
31. Progress counts only successes, so a failed run can never print
"31/31 downloaded". The manifest is validated for empty part keys,
negative sizes and colliding filenames, each of which would otherwise
corrupt a download silently rather than fail loudly.

Also gitignore /.wrangler/, which wrangler creates in whatever directory
it runs from and which carries the Cloudflare account id.
TestFetchFailsFast failed in CI on both platforms and reproduced locally
under -count=20. The test, not the fail-fast logic, was at fault: it
failed a single part and assumed that part's goroutine would run in the
first concurrency-sized batch. Go's scheduler does not guarantee that, so
on an unlucky run the failing part started only after siblings had
already completed a full 2s sleep, and the test counted their completions
as a fail-fast failure.

Instrumenting the handler showed the code was correct throughout: when
every part fails, 4 or 5 of 24 requests reach the server and Fetch
returns in 1-2ms, every run.

The test now fails every part, which is both the real-world shape it
guards (a 403 from the gate fails for every part) and deterministic.
Verified with -count=30.

Also tighten the dispatch itself: check ctx.Err() before racing for a
semaphore slot, since a select with both a free slot and a closed Done
channel picks at random.
@xe-nvdk
xe-nvdk merged commit 62fd212 into main Sep 9, 2026
3 checks passed
@xe-nvdk
xe-nvdk deleted the feat/sample-load branch September 9, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant