Skip to content

Flip the Explorer to the 202609 release - #359

Merged
rdhyee merged 2 commits into
isamplesorg:mainfrom
rdhyee:staging/explorer-flip-202609-rehearsal
Sep 12, 2026
Merged

rdhyee merged 2 commits into
isamplesorg:mainfrom
rdhyee:staging/explorer-flip-202609-rehearsal

Conversation

@rdhyee

@rdhyee rdhyee commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: close out the 202609 data release by pointing the live Explorer at it (the minimal wrap-up scope). What I did: finished the 9/9 mechanical rename into a reviewable change — new release manifest, hand-edited CANONICAL.md, docs that say which generation is live, and a fix for preload hints under a data-origin override — then checked it in a browser against the hash-verified 202609 files.

What this changes

  • Explorer (explorer.qmd) loads the isamples_202609_* files. It also covers the
    release-manifest filename and the search-index path.
  • Release manifest: isamples_202609_release_manifest.json is new, generated
    against the production keys with tools/build_release_manifest.py. The Explorer,
    tests/test_data_origin_contract.py and the data-origin-contract workflow path now
    use it. The 202608 manifest stays in _quarto.yml resources, so already-published
    links and cached older renders keep resolving.
  • CANONICAL.md, edited by hand:
    • §1 names the 202609 set, with search-index counts: 256 base + 769 hot shard files,
      1,030 objects.
    • §2's suffix history is left as it was; those rows are all 202608 files.
    • §3 marks 202608 superseded and 202609 live.
    • §4 documents the staging/ copies left on the bucket.
  • Preload hints: a small head script now adds the three boot-critical preloads, and
    skips them when ?data_base= or localStorage.ISAMPLES_DATA_BASE overrides the
    origin. Before, previewing a release from another prefix fired three prefetches that
    404'd against the default origin. On a local server the prefetch starts about 0.1 s
    later than with static tags (19 ms → 130 ms, one run each).
  • Docs: "what the Explorer serves today" notes now say 202609 (README,
    SERIALIZATIONS, how-to-use, data, tutorials index, why_h3, query-spec,
    DATA_PROVENANCE). Also updated: EXPLORER_QUERIES.md, the guided tour's data URLs,
    and the Explorer's own size/index figures (wide 300 → 237 MB; index 852 → 1,030
    objects). Manifest links now point at isamples.org, where the site deploys the file.
    The old data.isamples.org/isamples_202608_release_manifest.json link returned 404.
  • Left alone on purpose: 202608 references that are history or pipeline tooling —
    the comment on the one-off _v2 retrofit, FACET_HIERARCHY_PLAN.md,
    SEARCH_INDEX_V1.md, Makefile, scripts/, and the Worker deploy probes. Those
    probes use 202608 files, which stay served.

202609 vs 202608 — what's the same, what isn't

We compared the live 202608 files directly with the 202609 files.

  • Identical:
    • H3 clusters at res4/6/8 (38,462 / 112,019 / 176,669 cells): the same cells, with
      the same per-cell sample count, dominant source and source count. Centres differ
      by at most 0.000001°, which is rounding.
    • sample coordinates: 0 of 6,026,242 located samples moved
    • tree facet summaries and the tree cross-filter, row for row
    • located-sample index metadata: 6,026,242 samples, same per-source counts
    • the mineral ∪ soil tree union (333,253)
    • search results for the three ground-truth queries
  • Different:
    • Flat single-value material for 41,372 SESAR samples. Examples: mineral →
      biogenic non-organic material (12,584); soil → biogenic non-organic material
      (11,986); sediment → rock (7,144). Flat facet_summaries material counts move to
      match (e.g. soil 32,149 → 13,806), and about 110 facet_cross_filter rows change.
      In each generation, the summaries agree exactly with that generation's
      sample_facets_v4. The flat label is the first non-root material in each
      sample's list; sql_converter: byte-deterministic output; fix narrow edge row_id collisions pqg#26 made those lists keep source order instead of an
      arbitrary order. So 202609's label is stable where 202608's was arbitrary. Whether the
      first-listed material is the best single label has not been assessed. The default
      tree view is unaffected.
    • Place-name capitalisation for 11,114 located samples (OpenContext 8,457,
      SESAR 2,068, Smithsonian 393, GEOME 196), for example UTAH → Utah and
      3 Coconuts trail → 3 Coconuts Trail. All 11,114 are equal case-insensitively;
      no other place-name content changed.
    • The wide file is 236,995,193 B vs 300,303,095 B, with the same 170 row groups
      and ZSTD. It has 144 fewer rows: OpenContext location rows, from deterministic site
      de-duplication (pqg#26). We did not investigate why the file got smaller.

How it was checked

  • tools/verify_release.py check against the hash manifest: 1,046 files, 0 problems.
    This was run on the staged copy, and again on the production keys before this PR.
  • Local render (quarto render explorer.qmd, Quarto 1.8.24; CI pins 1.6.42), served
    with dev_server.py, loaded in Chrome against the 202609 files:
    • 38,462 clusters, 6,026,242 samples
    • SESAR 4,389,231 / OpenContext 1,104,985 / GEOME 291,210 / Smithsonian 240,816
    • __manifestCheck ok (only extra: offline-only df.parquet)
    • zero console errors
    • search totals: basalt 785, pottery Cyprus 1,305, axial seamount summit caldera 284
  • tests/test_data_origin_contract.py passes against the production origin with the
    new manifest.
  • Codex review: see the final comment on this PR.

Rollback

Revert this PR's merge commit (git revert -m 1 <merge sha>). The 202608 files on R2
were not modified, so the revert redeploys a fully working 202608 Explorer.

— isamples-0910 (Claude, for rdhyee)

🤖 Generated with Claude Code

https://claude.ai/code/session_012aY9Nvapmp5zMUPATZygb7

rdhyee and others added 2 commits September 10, 2026 11:25
Points the live Explorer, its release manifest, CANONICAL.md and the docs
that name the live generation at isamples_202609 (hash-verified on R2).

- explorer.qmd: every data URL, the manifest filename and the search-index
  path move to 202609. Preload hints are injected by a small head script and
  skipped under a data-origin override (?data_base= / localStorage), so
  staging previews no longer 404 against the default origin. Historical
  comments stay about 202608. The About section's sizes and search-index
  counts are re-derived from the 202609 manifest, and its manifest link now
  points at isamples.org.
- isamples_202609_release_manifest.json is added. The 202608 manifest stays
  deployed (_quarto.yml resources) so published links keep resolving.
  tools/build_release_manifest.py, tests/test_data_origin_contract.py and
  the data-origin-contract workflow follow 202609.
- CANONICAL.md, edited by hand:
  - section 1 names the 202609 set;
  - section 2's 202608 suffix history is unchanged;
  - section 3 marks 202608 superseded (live since isamplesorg#284) and records what
    202609 changes: 144 fewer location rows, a flat material change for
    41,372 SESAR samples via pqg#26 array ordering, and capitalisation-only
    place-name changes for 11,114 samples;
  - section 4 documents the staging copies.
- DATA_PROVENANCE.md: historical reproducibility/version-skew notes are
  scoped to their own builds.
- "What the Explorer serves" notes (README, SERIALIZATIONS, how-to-use,
  data, tutorials, query-spec), EXPLORER_QUERIES.md, the guided tour's data
  URLs, and the Playwright test comments move to 202609. The FTS ground
  truths were re-observed unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aY9Nvapmp5zMUPATZygb7
The 202609 publish set is now at the real top-level keys on
data.isamples.org, byte-verified (verify_release.py: 1046 ok, 0 problems).
tools/build_release_manifest.py was rerun with its default production base.
Only `base` and `generated_at_utc` change: sizes and ETags for all 21 files
match the staging-built manifest, and the search-index inventory is
unchanged. tests/test_data_origin_contract.py passes against the live
origin with this manifest (5 passed).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012aY9Nvapmp5zMUPATZygb7
@rdhyee rdhyee added the CC+Codex+LGTM Claude and Codex both reached LGTM; ready for human review label Sep 12, 2026
@rdhyee

rdhyee commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: have this flip reviewed by a second AI before it ships, as with the rest of the 202609 chain. What I did: ran three Codex review rounds over the branch diff and am posting the verdicts.

Round 1 — CHANGES REQUESTED, four P2 findings, all fixed in the branch:

  1. CANONICAL.md §3 understated the data change — the 202609 row now states that content is not identical and names both differences (flat material for 41,372 SESAR samples; 144 fewer location rows), plus what is unchanged.
  2. DATA_PROVENANCE.md carried reproducibility/version-skew statements that read as claims about 202609 — now scoped to the historical 202601/202604 builds, and flagged as not re-audited for the 202609 re-run.
  3. Deleting isamples_202608_release_manifest.json would have broken its published URL — the file is restored unchanged and both manifests stay in _quarto.yml resources.
  4. The recorded start of 202608's production life was wrong — corrected to the Production: OC sync (#272) + #277/#283 fixes → isamples_202608 #284 cutover (merged 2026-06-14), matching commit d26fabb.

Round 2 — LGTM. All four verified resolved; no new merge-blocking findings; 111 Node unit tests pass; runtime filenames, manifest entries and builder inventory agree. Codex noted its approval was conditional on the production promotion, the manifest regeneration and passing contract checks — all three have since been done and are described in the PR body.

Round 3 — LGTM, no findings. A narrow round over the later CANONICAL.md §3 edit (place-name capitalisation and the precise H3 wording). Codex reproduced the counts from the parquet files independently, and confirmed that the case-only place-name changes produce identical search-index token sequences, so search results are unaffected.

Claude + Codex: LGTM. Not merged — Raymond's call.

— isamples-0910 (Claude, for rdhyee)

@rdhyee
rdhyee merged commit 8faccb8 into isamplesorg:main Sep 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CC+Codex+LGTM Claude and Codex both reached LGTM; ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant