Flip the Explorer to the 202609 release

.github/workflows/data-origin-contract.yml

@@ -10,7 +10,7 @@ on:
     paths:
       - 'workers/data-isamples-org/**'
       - 'tests/test_data_origin_contract.py'
-      - 'isamples_202608_release_manifest.json'
+      - 'isamples_202609_release_manifest.json'
       - '.github/workflows/data-origin-contract.yml'
   schedule:
     - cron: '17 6 * * *'   # daily 06:17 UTC

CANONICAL.md

@@ -6,33 +6,42 @@ accumulation is superseded versions and orphan snapshots around it. This file
 names the canonical set and documents every non-canonical survivor so the
 accident is neutralized even where files remain on the bucket.*
 
+*Updated 2026-09 for the 202608 → 202609 generation flip: §1 now names the 202609
+set; §2's rows are unchanged history (all of them are 202608 files); §3 records
+202608 as superseded.*
+
 **Companion:** the machine-readable release manifest
-(`tools/build_release_manifest.py` → `isamples_202608_release_manifest.json`)
+(`tools/build_release_manifest.py` → `isamples_202609_release_manifest.json`; the
+superseded `isamples_202608_release_manifest.json` stays deployed for old links)
 enumerates the same canonical set with sizes/checksums; the Explorer cross-checks
 it at boot (#334 v0). A file absent from the manifest is non-canonical **by
 definition**.
 
-## 1. The canonical (live) set — 202608
+## 1. The canonical (live) set — 202609
 
 Exactly what production `explorer.qmd` loads, all under `https://data.isamples.org/`:
 
 | File | Role (plain English) |
 |---|---|
-| `isamples_202608_wide.parquet` | Full sample detail, one row per entity — everything derives from this |
-| `isamples_202608_samples_map_lite_v3.parquet` | Slim map/table columns (coords, label, place, date, h3 cells) |
-| `isamples_202608_sample_facets_v4.parquet` | Per-sample facet URIs + search-text blob (the `?fts=off` scan target) |
-| `isamples_202608_h3_summary_res{4,6,8}.parquet` | Pre-counted globe clusters at 3 zoom tiers |
-| `isamples_202608_facet_summaries.parquet` | Facet checkbox counts, no filters active |
-| `isamples_202608_facet_cross_filter.parquet` | Facet counts under single cross-filter |
-| `isamples_202608_facet_tree_summaries.parquet` | Hierarchical (tree) facet counts |
-| `isamples_202608_facet_tree_cross_filter.parquet` | Tree facet counts under cross-filter |
-| `isamples_202608_sample_facet_membership.parquet` | Sample↔facet-tree membership |
-| `isamples_202608_sample_facet_masks.parquet` | Bitmask substrate for multi-filter counts |
-| `isamples_202608_facet_node_bits.parquet` | Facet-node bit assignments for the masks |
-| `isamples_202608_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) |
-| `isamples_202608_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) |
-| `vocab_labels_202608.parquet` | URI → human label (539 entries; verified complete 2026-07-22) |
-| `isamples_202608_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 base token shards and ~591 hot-token sub-files (852 objects total incl. sidecars; base inventory in `shard_sizes.json`, totals in `build_stats.json`); `df.parquet` is offline-only |
+| `isamples_202609_wide.parquet` | Full sample detail, one row per entity — everything derives from this |
+| `isamples_202609_samples_map_lite_v3.parquet` | Slim map/table columns (coords, label, place, date, h3 cells) |
+| `isamples_202609_sample_facets_v4.parquet` | Per-sample facet URIs + search-text blob (the `?fts=off` scan target) |
+| `isamples_202609_h3_summary_res{4,6,8}.parquet` | Pre-counted globe clusters at 3 zoom tiers |
+| `isamples_202609_facet_summaries.parquet` | Facet checkbox counts, no filters active |
+| `isamples_202609_facet_cross_filter.parquet` | Facet counts under single cross-filter |
+| `isamples_202609_facet_tree_summaries.parquet` | Hierarchical (tree) facet counts |
+| `isamples_202609_facet_tree_cross_filter.parquet` | Tree facet counts under cross-filter |
+| `isamples_202609_sample_facet_membership.parquet` | Sample↔facet-tree membership |
+| `isamples_202609_sample_facet_masks.parquet` | Bitmask substrate for multi-filter counts |
+| `isamples_202609_facet_node_bits.parquet` | Facet-node bit assignments for the masks |
+| `isamples_202609_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) |
+| `isamples_202609_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) |
+| `vocab_labels_202609.parquet` | URI → human label (539 entries, same count as 202608) |
+| `isamples_202609_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 base token shards and 769 hot-token sub-files (1,030 objects total incl. sidecars; base inventory in `shard_sizes.json`, totals in `build_stats.json`); `df.parquet` is offline-only |
+
+The `_v3`/`_v4` suffixes carry over from 202608 so the Explorer's file names stay
+recognizable; 202609 was built with those fixes natively and has **no** earlier
+`_v2`/`_v3` siblings on the bucket.
 
 The ~9-file facet family looks baroque but is load-bearing: it is the price of
 fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each
@@ -64,14 +73,17 @@ boot; this table makes them legible to humans.)
 | `isamples_202601_*` | **Archival** — the coherent snapshot in Zenodo draft 21288719 |
 | `isamples_202604_*` | **NOT orphan — backs the stable `/current/wide.parquet` alias** used by data.qmd and multiple tutorials (see SERIALIZATIONS.md). Must NOT be moved until the alias is repointed and consumers audited |
 | `isamples_202606_*` | Pipeline intermediate (the #272 OC-concept-enriched wide that fed 202608 — see DATA_PROVENANCE.md step 3b). Not loaded by anything at runtime; safe to attic post-grant, lineage documented |
-| `isamples_202608_*` | **Live** (production Explorer) |
+| `isamples_202608_*` | **Superseded by 202609** (live from the #284 cutover, merged 2026-06-14, until the 202609 flip). Still served, bytes unchanged, so old links and notebooks keep working. Not loaded by the Explorer. Its §2 suffix history stays valid |
+| `isamples_202609_*` | **Live** (production Explorer). A deterministic rebuild with the same sample identities and counts as 202608 — but **not identical content**: (a) 144 fewer OpenContext location rows, because site de-duplication now picks its winner deterministically (pqg#26); (b) the flat single-value `material` differs for 41,372 SESAR samples, because the flat value is the first non-root entry of `p__has_material_category` (see DATA_PROVENANCE.md, "Material selection") and pqg#26 made that array input-ordered instead of arbitrary. So flat `facet_summaries` material counts moved (e.g. soil 32,149 → 13,806) and ~110 `facet_cross_filter` rows changed; (c) `place_name` differs for 11,114 located samples in capitalisation only (e.g. `UTAH` → `Utah`; all 11,114 are equal case-insensitively). Unchanged: tree facet summaries and tree cross-filter (row for row); sample coordinates (0 of 6,026,242 moved); H3 cells with the same per-cell sample count, dominant source and source count (centres within 0.000001°, rounding); located-sample and per-source counts |
 
 ## 4. Conveniences (not load-bearing; fine to keep, labeled)
 
 | File | Note |
 |---|---|
 | `isamples_202601_wide_h3.parquet` | Analyst convenience (wide + precomputed H3 cells). In the Zenodo deposit; NOT loaded by the Explorer |
 | `*.csv` twins of lite/H3 (~640 MB) | Convenience exports; parquet is authoritative; excluded from Zenodo by design |
+| `staging/isamples_202609/*` | Pre-cutover rehearsal copy of the §1 set (hash-verified against the build). NOT loaded by the Explorer; safe to delete once the real keys are verified |
+| `staging/isamples-0909-smoketest/*` | Upload-tool smoke test (4 small files). Not referenced anywhere; safe to delete |
 
 ## 5. Post-grant simplification options (deliberately NOT done in-grant)
 
@@ -81,5 +93,6 @@ boot; this table makes them legible to humans.)
    9 files trade storage (cheap) for browser CPU (scarce); collapsing them is an
    engineering project, not cleanup.
 3. If a fresher export ever lands (#320), cut ONE coherent generation and retire
-   202608 wholesale via the manifest.
+   the then-live generation wholesale via the manifest. (202609 is not that
+   fresher cut — it rebuilt the same sample records reproducibly.)
 <!-- cc:2026.07.23 -->

DATA_PROVENANCE.md

@@ -4,19 +4,20 @@ How the explorer's derived parquet files are generated, from root to publish.
 **Not exhaustive as of 2026-08-05 — see the coverage caveat below.**
 *Reviewed 2026-06-02 (CC, via codebase audit). Complements `SERIALIZATIONS.md` (format/schema reference); this file is the end-to-end build chain + the automation gaps.*
 
-> ⚠️ **Coverage caveat (2026-08-05).** The DAG below documents the **seven-file
+> ⚠️ **Coverage caveat (2026-08-05; live generation updated 2026-09).** The DAG below documents the **seven-file
 > derived substrate as of the 2026-06-02 review**. It does *not* cover the whole
-> live `202608` family. Known omissions:
+> live `202609` family. Known omissions:
 >
 > - `sample_facet_masks`, `facet_node_bits`, `sample_facet_index`,
 >   `sample_facet_index_meta` (the bitmask count path, #299/#304/#305/#313)
 > - `sample_facet_membership`
 > - `facet_tree_summaries`, `facet_tree_cross_filter` (the tree facet path, #290)
-> - the sharded search index `isamples_202608_search_index_v1/` (#171)
+> - the sharded search index `isamples_202609_search_index_v1/` (#171)
 >
-> The *build chain and the automation gaps* it describes are still accurate for
-> the files it does cover; treat it as incomplete rather than wrong. Authoritative
-> current inventory: `isamples_202608_release_manifest.json` / `CANONICAL.md`.
+> The *build chain and the automation gaps* it describes were accurate, as of that
+> review, for the files it covers. They have not been re-audited for the 202609
+> re-run, so treat this file as incomplete and dated rather than wrong. Authoritative
+> current inventory: `isamples_202609_release_manifest.json` / `CANONICAL.md`.
 
 > **Load-bearing constraint:** the **root export cannot be regenerated.** It was produced from the iSamples Central Solr API (`central.isample.xyz`), **offline since Aug 2025**. The Zenodo-archived export is a **frozen root**. Any *new* data (e.g. concept URIs, thumbnails) therefore must come from a **per-source supplementary file merged into the base by `pid`** — the "sidecar" pattern (see Stage 3) — not from re-exporting.
 
@@ -87,11 +88,15 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap
 
 > ⚠️ **Snapshot note (2026-08-05).** The version-skew bullets below were written
 > when the deployed derived files were `202601` and the wide was `202604`. That is
-> no longer the live state: the Explorer now serves the **`202608`** family
-> (`sample_facets_v4`, `samples_map_lite_v3`, `wide` at 300,303,095 B). The
-> *reproducibility* gap the bullets describe is still real and still unresolved —
-> only the version numbers in them are historical. Authoritative current inventory:
-> [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json),
+> no longer the live state: the Explorer now serves the **`202609`** family
+> (`sample_facets_v4`, `samples_map_lite_v3`, `wide` at 236,995,193 B). The
+> reproducibility and version-skew bullets below describe **those historical builds**
+> (the `202601` derived files and the `202604` wide) and are kept as a record. The
+> 202609 generation came from a full re-run of the chain with per-file hashes recorded
+> (`tools/verify_release.py`; toolchain pin in #358). These bullets have not been
+> re-audited against that re-run, so don't read them as statements about 202609.
+> Authoritative current inventory:
+> [`isamples_202609_release_manifest.json`](https://isamples.org/isamples_202609_release_manifest.json),
 > human twin `CANONICAL.md`.
 
 - ⚠️ **The deployed `202601` derived files are NOT reproducible** from any available wide. A rebuild yields **528,983** root-material rows (pre-#271); the deployed `sample_facets_v2` has **346,768** — so the live files came from a different/unrecorded Stage-4 process, *and* the data has since rolled (wide is now `202604`). Treat a fresh `build_frontend_derived.py` run as the new source of truth, not as a bit-for-bit reproduction of the deployed files.

EXPLORER_QUERIES.md

@@ -23,22 +23,22 @@ server. You can open any of these URLs directly, or point DuckDB at them
 one place, `explorer.qmd` around **line 800-864**, e.g.:
 
 ```js
-lite_url    = `${R2_BASE}/isamples_202608_samples_map_lite_v3.parquet`   // map points + table
-wide_url    = `${R2_BASE}/isamples_202608_wide.parquet`                  // full sample detail
-facets_url  = `${R2_BASE}/isamples_202608_sample_facets_v4.parquet`      // material/context/object_type + search text
-h3_res4_url = `${R2_BASE}/isamples_202608_h3_summary_res4.parquet`       // pre-counted globe dots (world zoom)
+lite_url    = `${R2_BASE}/isamples_202609_samples_map_lite_v3.parquet`   // map points + table
+wide_url    = `${R2_BASE}/isamples_202609_wide.parquet`                  // full sample detail
+facets_url  = `${R2_BASE}/isamples_202609_sample_facets_v4.parquet`      // material/context/object_type + search text
+h3_res4_url = `${R2_BASE}/isamples_202609_h3_summary_res4.parquet`       // pre-counted globe dots (world zoom)
 ```
 
 | File | Plain-English role | Roughly how big |
 |---|---|---|
-| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~300 MB |
+| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~237 MB |
 | `..._samples_map_lite_v3.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~63 MB |
 | `..._sample_facets_v4.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~69 MB |
 | `..._h3_summary_res{4,6,8}.parquet` | Pre-counted dots for the globe at 3 zoom tiers (continent / region / neighborhood), so zooming out never counts 6M rows live | tiny–few MB |
 | `..._facet_summaries.parquet`, `..._facet_cross_filter.parquet`, `..._facet_tree_*.parquet` | Pre-computed facet-checkbox counts at various levels of "how many filters are active" — the whole point of these is to avoid a live COUNT over millions of rows | KB–tens of MB |
 | `..._sample_facet_masks.parquet`, `..._sample_facet_index.parquet` | Bitmask tricks so 2+ facet filters at once are still fast (see `SERIALIZATIONS.md` §4.12 if you want the gory detail) | ~10 MB each |
 | `vocab_labels_*.parquet` | URI → human-readable label lookup (e.g. `.../material/1.0/rock` → "Rock") | ~60 KB |
-| `..._search_index_v1/` (852 files) | Pre-built search index (like a book's back-of-book index, sharded): token shards + tiny sidecars (`hot_tokens.json`, `df.parquet`, `build_stats.json`). The default search path since 2026-07-17 | few KB–few MB per shard |
+| `..._search_index_v1/` (1,030 files) | Pre-built search index (like a book's back-of-book index, sharded): token shards + tiny sidecars (`hot_tokens.json`, `df.parquet`, `build_stats.json`). The default search path since 2026-07-17 | few KB–few MB per shard |
 
 *Full list with exact schemas: `SERIALIZATIONS.md`. This table is the subset
 that matters for "what happens when I click around the Explorer."*
@@ -87,7 +87,7 @@ of filters" doesn't pre-aggregate cleanly.
 
 **Default path (since 2026-07-17): a pre-built search index.** Your query is
 split into tokens, and each token maps (by hash) to a small parquet shard of
-a pre-built inverted index (`isamples_202608_search_index_v1/`, 852 files —
+a pre-built inverted index (`isamples_202609_search_index_v1/`, 1,030 files —
 think "the index at the back of a book, one file per drawer"). The browser
 fetches only the few KB-to-MB shards for *your* tokens, intersects the
 matching sample ids, and ranks them by relevance (BM25 — the standard
@@ -151,7 +151,7 @@ LEFT JOIN read_parquet('vocab_labels.parquet') mat_lbl ON mat_lbl.uri = mat.pid
 WHERE s.pid = '<clicked pid>'
 ```
 This is the one query that reads from `wide.parquet` on click (everything
-above deliberately avoids touching the 300 MB wide file until you actually
+above deliberately avoids touching the 237 MB wide file until you actually
 need full detail on one sample).
 
 ## Try it yourself
@@ -162,15 +162,15 @@ You don't need the browser — any of this works from the DuckDB CLI or
 ```sql
 -- how many samples per source, right now, live off the public URL
 SELECT n AS source, COUNT(*)
-FROM read_parquet('https://data.isamples.org/isamples_202608_wide.parquet')
+FROM read_parquet('https://data.isamples.org/isamples_202609_wide.parquet')
 WHERE otype = 'MaterialSampleRecord'
 GROUP BY n ORDER BY 2 DESC;
 
 -- the "pottery" search the Explorer's FALLBACK path (?fts=off) runs;
 -- the default path since 2026-07-17 probes the sharded search index instead
 -- (JS, not a single SQL statement — see SEARCH_INDEX_V1.md)
 SELECT pid, label, source
-FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v4.parquet')
+FROM read_parquet('https://data.isamples.org/isamples_202609_sample_facets_v4.parquet')
 WHERE description ILIKE '%pottery%'
 LIMIT 20;
 ```

README.md

@@ -3,14 +3,14 @@ title: isamples.github.io
 subtitle: README for the isamples.github.io source
 ---
 
-> ⚠️ **Snapshot note (2026-08-05).** The **`202601`** file examples below pin a
+> ⚠️ **Snapshot note (2026-08-05; live generation updated 2026-09).** The **`202601`** file examples below pin a
 > stable, versioned snapshot. Those objects still exist and their byte counts are
 > correct, but they are **not what the Interactive Explorer serves today** — the
-> live app reads the **`202608`** family, including `sample_facets_v4`,
+> live app reads the **`202609`** family, including `sample_facets_v4`,
 > `samples_map_lite_v3`, `wide`, and the sharded search index. (Other generations
 > may also appear here, such as the `current/` alias or the `202512` narrow file.)
 > Use `202601` for a stable citable snapshot; for what the Explorer actually
-> loads, see `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`).
+> loads, see `isamples_202609_release_manifest.json` (human twin: `CANONICAL.md`).
 
 # isamplesorg.github.io
 

SERIALIZATIONS.md

@@ -7,14 +7,14 @@ toc: true
 categories: [data, architecture, parquet]
 ---
 
-> ⚠️ **Snapshot note (2026-08-05).** The **`202601`** file examples below pin a
+> ⚠️ **Snapshot note (2026-08-05; live generation updated 2026-09).** The **`202601`** file examples below pin a
 > stable, versioned snapshot. Those objects still exist and their byte counts are
 > correct, but they are **not what the Interactive Explorer serves today** — the
-> live app reads the **`202608`** family, including `sample_facets_v4`,
+> live app reads the **`202609`** family, including `sample_facets_v4`,
 > `samples_map_lite_v3`, `wide`, and the sharded search index. (Other generations
 > may also appear here, such as the `current/` alias or the `202512` narrow file.)
 > Use `202601` for a stable citable snapshot; for what the Explorer actually
-> loads, see `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`).
+> loads, see `isamples_202609_release_manifest.json` (human twin: `CANONICAL.md`).
 
 ## 1. Purpose and scope
 

_quarto.yml

@@ -2,6 +2,9 @@ project:
   type: website
   output-dir: docs
   resources:
+    - isamples_202609_release_manifest.json
+    # Superseded release's manifest stays deployed so already-published links and
+    # cached older Explorer renders keep resolving (CANONICAL.md §3).
     - isamples_202608_release_manifest.json
     - assets/js/source-palette.js
     - assets/js/sql-builders.js