feat(sample): arcli sample {list,show,load} for real datasets - #26
Merged
Merged
Conversation
added 2 commits
September 8, 2026 18:41
Load a real public dataset into your own Arc instead of writing one synthetic point and wondering what Arc is for. `arcli sample load citibike` downloads a month of NYC Citi Bike trips, verifies every file against the published SHA-256, imports it, and prints two queries worth running against what just landed. - `sample list` — datasets and months, one row per loadable pair - `sample show` — details, licence, time window; `-o json` prints every file URL and checksum so the data can be fetched without arcli and imported with `arcli import parquet` - `sample load` — download, verify, import; creates the target database, resumes an interrupted run, and asks before adding to a measurement that already holds rows Downloads are fetched from samples.basekick.net, the only Basekick- controlled host arcli contacts. They carry the same User-Agent and Arcli-Installation-Id as every other arcli request, honouring the existing DO_NOT_TRACK and send_installation_id opt-outs, and never the token, the Arc endpoint, or anything about the user's data. The README privacy section said arcli never contacts Basekick; that is no longer true and now says what is actually sent. Fetching fails fast: a failure cancels the remaining downloads, so an unrecognised build that 403s on every part costs 4 requests rather than 31. Progress counts only successes, so a failed run can never print "31/31 downloaded". The manifest is validated for empty part keys, negative sizes and colliding filenames, each of which would otherwise corrupt a download silently rather than fail loudly. Also gitignore /.wrangler/, which wrangler creates in whatever directory it runs from and which carries the Cloudflare account id.
TestFetchFailsFast failed in CI on both platforms and reproduced locally under -count=20. The test, not the fail-fast logic, was at fault: it failed a single part and assumed that part's goroutine would run in the first concurrency-sized batch. Go's scheduler does not guarantee that, so on an unlucky run the failing part started only after siblings had already completed a full 2s sleep, and the test counted their completions as a fail-fast failure. Instrumenting the handler showed the code was correct throughout: when every part fails, 4 or 5 of 24 requests reach the server and Fetch returns in 1-2ms, every run. The test now fails every part, which is both the real-world shape it guards (a 403 from the gate fails for every part) and deterministic. Verified with -count=30. Also tighten the dispatch itself: check ctx.Err() before racing for a semaphore slot, since a select with both a free slot and a closed Done channel picks at random.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
arcli sample list | show | loadso a new user can put real data into their own Arc instead of writing one synthetic point and wondering what Arc is for.sample load citibikedownloads a month of NYC Citi Bike trips (31 files, 39.4 MiB, 1,003,770 rows), verifies every file against the published SHA-256, imports it, and prints two queries worth running against what just landed.sample show citibike -o jsonprints every file URL and checksum, so anyone who would rather not fetch from us can download the data themselves and usearcli import parquet.samples.basekick.net, the only Basekick-controlled host arcli contacts. Requests carry the sameUser-AgentandArcli-Installation-Idas every other arcli request (the existingDO_NOT_TRACK/send_installation_idopt-outs apply) and never the token, the Arc endpoint, or anything about the user's data.Behaviour worth knowing
--yesskips it, and a non-TTY refuses rather than hanging).--download-onlyneeds no Arc connection at all.Review findings fixed before this PR
The internal deep review found two things that mattered, both fixed here:
Fetchhad no fail-fast. An unrecognised build 403s on every part, so a user would have waited through 31 failures. A failure now cancels the rest: 4 requests instead of 31. Regression-tested.31/31 downloadedon a run that then failed, because the counter incremented on errors too. It now counts only successes.Also fixed: a goroutine leak on early return (stragglers could write into a directory the command had already deleted), manifest validation for empty part keys / negative sizes / colliding basenames, three independent timeouts converged onto
--timeout, and a generic query fallback so a future dataset does not silently ship without the payoff.One process note: my first fail-fast test passed while the code was broken and failed once it was fixed — it was measuring its own sleep, not the cancellation. Caught only by running it. Both regression tests are in the permanent suite.
Test plan
gofmt -l .,go vet ./...,go test -race -count=1 ./...— all cleanarcli sample load citibikeagainst a localarc serve: 31/31 files, checksums verified, 1,003,770 rows imported in ~3sarcli query— correct results (top stationW 21 St & 6 Ave, 4,070; commute curve peaks 08:00 / 17:00)--download-onlywith no Arc configured;--download-dirre-run reports 31 cached, zero re-downloads--tokenwithout--endpoint,--endpointonly (auth disabled),ARC_ENDPOINT,-cmissing connection,-o arrow,DO_NOT_TRACK=1Authorizationnever sent to the sample host