Simplify fiboa publish for catalog-driven publication - #214
Open
ivorbosloper wants to merge 104 commits into
Open
Simplify fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 104 commits into
fiboa publish for catalog-driven publication#214ivorbosloper wants to merge 104 commits into
Conversation
Co-authored-by: Matthias Mohr <matthias@mohr.ws>
# Conflicts: # CHANGELOG.md # pixi.lock
…easing memory requirements for large data sets
`fiboa publish` now only converts, validates, builds PMTiles and writes a collection.json with relative links, file:size/file:checksum (multihash), a web-map-links v1.3.0 `pmtiles` link with `pmtiles:layers` and a `visual` asset. README/LICENSE generation, the data-survey lookup and the S3 upload are gone; catalogs such as fieldsoftheworld/harmonized-field-data-catalog own those. spdx-license-list is only needed by tests and moves to the dev feature. Also: - FiboaBaseConverter: keep the determination:datetime column that `use_variant_as_determination` adds; it was removed again as unlisted (affected dk, hr). - be_vlg: drop plots without a crop code (one in 2023 failed validation), take the determination date from the variant year, add the 2026 edition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ivorbosloper
changed the title
fiboa publish for catalog-driven publication
…note Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fallbacks vecorel/cli#23 is merged and released, so the converter output arrives pre-sorted; fiboa_cli/conversion/hilbert.py and the try/except fallback imports are gone. The in-place resort now writes without the embedded arrow schema: the large_binary widening used for >2GB takes must not leak into the file, or a rewritten part stops schema-matching untouched pre-sorted siblings during the streaming merge (masked before 0.2.16, when every part needed rewriting). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… write store_schema=False silently drops ALL key-value metadata (geo, collection), not just the embedded arrow schema — the first production resort produced a file geopandas could not read. Write through a ParquetWriter against the original narrow schema instead, casting each sub-2GB batch back from the widened types; regression test asserts the resorted file keeps its metadata and narrow schema. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rByFields Cantabria's 2010-2014 SIGPAC layers are server-side joins: every field is table-qualified, so where=OBJECTID>x returned 'Failed to execute query'. Probe the layer for the real key name and page on it. Also 'sortBy' is not an ArcGIS parameter (orderByFields is); it only worked on layers whose default order happens to be the key. es_cb strips the table prefixes after download, first occurrence wins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vecorel.org/fiboa.org fail intermittently; a blip at write time killed conversions after 25-minute source downloads (four times this week). The needed URIs are known up front, so fetch them first — retrying with backoff, failing fast and cheap — and let load_file's per-process cache make the write itself network-free. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
I'm not really involved in the publishing (yet?), so I can't really judge this. But this PR contains a lot of unrelated changes, so I suspect we need to merge something else first to get this easier to review. |
orderByFields costs ~100 s per request on joined layers (the server sorts the whole join every page), which made the Cantabria 2010-2014 layers take days. A range filter on the unique key answers in ~1 s and a window of page_size ids cannot overflow a page, so fetch min/max once (two sorted one-row queries) and page by half-open id windows, skipping empty ones. Pages cached by the old scheme are reused when their ids prove they cover a window exactly. A where filter in rest_params is now ANDed in instead of being clobbered by the paging clause. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
…y variant ideg.xunta.gal serves SIXPAC_2014..SIXPAC_2026 (2010-2013 never existed, 2025/2026 were missing). The older campaigns differ: 2014 names the layer RECINTO with SUP_SIGPAC and no DN_OID, 2015 uses SUP_SIX/USO_SIX, 2020 lacks DN_OID but has IDGEOM. Rename per page in file_migration so the land-use filter and id checks see the canonical names; 2014 gets the SIGPAC recinto reference as id, 2020 the geometry id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
Every SIXPAC_<year> MapServer of es_ga has its Recintos layer at id 2, so pages of different years collided in the cache and later years silently re-read the first year's pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
~16k pages of 1000 features are concatenated per edition; the 20 unmapped attribute columns would otherwise stay in memory until the very end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvUrMu5mhX3WEUoLLkm8TG
A null or empty geometry survives conversion but breaks the canonical Hilbert sort at the very last step (geopandas refuses hilbert_distance on such a GeoSeries), so a 90-minute read ends with nothing written. es_ga 2020-2022 each failed this way; the rate is about 1 in 250,000 features. Drop those rows under the same bounded max_dropped_share rule already used for the required non-null properties, so a converter that produces many of them still errors out. Also widen the schema prewarm budget from 5 attempts (~30 s) to 8 with a 60 s cap (~4 min): a vecorel.org blip outlasted the old budget and killed es_ga 2018 and 2019 outright. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSwkoiouWUsspkFd9aQ2AZ
Galicia had no MT (matorral) code until 2023: scrub was coded PR, which the Spanish base filter keeps as grazing land. Sampling the cached pages shows PR at ~23% of features for every edition 2014-2022 and MT entirely absent, then the two swap from 2023 on (MT ~23%, PR ~1%). Keeping PR therefore left the pre-2023 editions about 75% larger than 2023+ (8.76M vs 4.97M fields for es_ga 2022 vs 2023) with no change on the ground. Exclude PR for campaigns before 2023 so the published series is comparable; 2023+ is untouched and still keeps the genuine shrub pasture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSwkoiouWUsspkFd9aQ2AZ
IFAP restructured the file for 2025: the field boundaries moved from "Culturas_<district>" layers into "T<NUTS 3 code>" ones (beside an empty "Culturas" container and a "Codes" lookup table), the crop code column is now PUN_CUL_CO, the crop name is gone, and the file is published in WGS 84 with Shape_Area and Shape_Length still computed in the source units -- degrees, so 2.7e-07 where 2023 says 3046 m2. Widen the layer filter to both shapes, rename the crop code back in migrate(), and recompute both metrics on EPSG:6933 when the file is geographic, the same projection `fiboa improve` uses to fill missing sizes. Take the determination date from the variant instead of the hard-coded 2023, now that a second edition exists; for 2023 that yields the same value it published before. Editions of one converter drift apart exactly like this, so the test list now accepts a "<id>#<label>" entry that runs a second edition from the same folder of input files, and pt is pinned to 2023 with pt#2025 covering the new shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Both were left as "Spain " with the region missing, which is what names every per-edition item in the catalog: "Spain 2026 (GeoParquet)", two spaces and no way to tell Galicia from Castilla-La Mancha. Their titles already carry the region, so take it from there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The converter left `provider = ""`, so the collection credited only Fields of
the World as processor and host, and rashid flagged it: no provider carried the
'producer' role, and the collection had no link back to the publisher. The
attribution string already names them ("©Government of Cantabria. Free
information available at https://mapas.cantabria.es"); take the provider from
there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The provider URL was https://gobiernoabierto.navarra.es/ — Navarra's open government portal, on a converter for the Balearic Islands. The name beside it and the source comments both say Govern de les Illes Balears, whose site is caib.es. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
`sources` was the bare Zenodo URL, so GDAL was handed the zip itself and answered "not recognized as being in a supported file format". Its three siblings (ec_lt, ec_lv, ec_si) all name the shapefile inside their archive; do the same here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Both converters declare a column required that the 2021 EuroCrops release does
not always fill, so writing the GeoParquet fails on the whole dataset:
Field pyarrow.Field<EC_NUTS3: string not null> was non-nullable but pandas
column had 1 null values
One Latvian parcel has no EC_NUTS3; four Slovenian ones have no Latin crop name
(crop_lat_e). The tests never caught it because the fixtures are the first 100
rows, where both columns are filled.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
column_migrations named JAHR — German, copied from de_sax — so it never ran
("Column 'JAHR' not found in dataset, skipping migration") and the bare year
reached the STAC writer as determination:datetime: "Invalid isoformat string:
'2021'". Estonia's column is taotlusaas, the shapefile's ten-character
truncation of taotlusaasta.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
Slovenia: four of the 828,161 parcels in the 2021 release carry no crop at all — SIFRA_KMRS, RASTLINA, CROP_LAT_E, COLOR and the EC_* columns are all empty for them, and three parcels lack EC_NUTS3. Requiring any of those fails the write for the whole dataset, one column at a time. Only the GERK parcel id is always present, so that is all the schema requires now. Checked against the source rather than guessed, after the same failure recurred on the next column. Estonia: taotletud_kultuur was required and typed but no column ever produces it — the crop code is mapped to crop:code — so the custom schema described a property that cannot exist. Dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
… zip Each edition ships as a zipped GeoPackage, and the variants named the archive alone, so GDAL opened it as /vsizip/...gpkg.zip. A GeoPackage is SQLite, whose reads are random access, and every seek re-inflates the deflate stream from the start: converting the 2018 edition read 85 GB out of the 1.7 GB archive in 40 minutes and had still not finished reading the layer. Naming the file inside the archive (as at_block already does) makes vecorel extract it once into cache/extracted.*/ and read a plain file. The glob keeps it working across the editions, whose inner names differ. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
…a of use
Austria's 2018 edition carries its CRS as a bare projection definition — an
unnamed "MGI / Austria Lambert" that pyproj cannot match to an EPSG code — so
crs_total_bounds raises and `fiboa publish` died after a five-minute conversion
with "Cannot derive total bounds for projected CRS {...}".
The CRS-derived grid exists so that separately converted parts of one dataset
sort into the same order. Where the CRS cannot supply it, the dataset's own
bbox orders the file just as well; only that mergeability is lost. Wrap it in
hilbert_reference_bounds(crs, fallback) and use it from all three call sites
(publish, the per-file merge, the DuckDB converter). Publish warns and skips
ordering when neither is available, rather than losing the whole conversion —
the file is valid either way, just slower to filter spatially.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The converter mapped FLIK and SCHLAGNR to their own columns but never produced `id`, so the collection shipped without the one column every other collection in the catalog has, and catalogize's spatial sample failed with "Referenced column id not found in FROM clause". There is no natural key to use: the 2025 shapefile holds 880,581 features with only 874,222 distinct (FLIK, SCHLAGNR) pairs, so neither the field block nor the pair identifies a parcel. The row index does, per edition, which is what hr does for the same reason; FLIK and SCHLAGNR stay as attributes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
…nverter Both were published without `id`. sk shows why it can happen quietly: it sets `index_as_id = True`, which fills the column, but `columns` never named `id`, and the base converter's "remove unlisted columns" step took it straight back out. de_bb never had either half. Neither source has a natural key. Brandenburg's ref_ident is a field block reference and gp_xx repeats across rows; Slovakia's KODKD is an LPIS block code shared by several fields and sometimes empty. Both take the row index, as de_nds and hr do, keeping the source codes as their own attributes. Nothing downstream catches this — a file without `id` validates — so the base converter now refuses to run without a mapping, naming the fix in the message, and a test walks every converter to keep it that way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The converter claimed CC-BY-NC-4.0, justified by two comments that belong to es_ib: they point at caib.es and quote the Balearic Islands' terms. Cantabria publishes under neither. Its policy (Decreto 87/2013, modified by Decreto 102/2018) defines two licences of its own, both free of charge — a non-commercial one and a commercial one, the latter needed only for reselling the data — so "NC" also overstates the restriction: their FAQ allows indirect economic benefit under the free non-commercial licence. The Esri service the converter reads names no licence at all, only "Gobierno de Cantabria-FEGA" as the copyright holder. Name the licence and link its terms, and use the attribution wording the licence itself prescribes, verbatim, instead of a translation of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
The column was mapped to farmer_id, but its values are 16-character FLIKs: DE + BB + LI (Landwirtschaft/InVeKoS) + two digits for the capture year + two for the district + six sequential, e.g. DEBBLI0268030574 — the same structure de_nds and de_nds_block publish as `flik` (DENILI1742040045). Brandenburg's own description of the Feldblockkataster spells that layout out. So the collection was telling readers a field block reference identified the applicant. Map it to `flik` with the flik extension, as its neighbours do, which also drops the custom schema this converter had to carry for the invented property. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AL2Yj4GXtEkdfwVz24Mv1U
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
fiboa publishnow only converts, validates, builds PMTiles and writes acollection.jsonwith relative links,file:size/file:checksum(sha256 multihash), a web-map-links v1.3.0pmtileslink withpmtiles:layers, and avisualasset. README/LICENSE generation, the data-survey lookup and theaws s3 syncupload are removed;--generate-meta,--data-url,--s3-upload-path,--yes,--editor,--converted-by,--data-survey-urlare gone,--no-pmtilesand--tippecanoe-optsare new. README and CHANGELOG updated.spdx-license-listis only used by tests and moves to the pixidevfeature.FiboaBaseConverter: thedetermination:datetimecolumn added byuse_variant_as_determinationwas dropped again by the base converter's "remove unlisted columns" step; it is now listed incolumnsautomatically (affected dk, hr).be_vlg: drop plots without a crop code (one plot in the 2023 edition failed validation), derive the determination date from the variant year instead of a constant, add the 2026 edition.Why
Publishing moves to the git-backed Portolan catalog at https://github.com/fieldsoftheworld/harmonized-field-data-catalog (source.coop/ftw/harmonized-field-data), which owns the catalog layer (styles, thumbnails, AGENTS.md/README, uploads). fiboa-cli stays the single home of conversion logic and produces the per-dataset building blocks.
Verification
The three pilot collections built with this branch pass
rashid check --livewith 0 errors against the published catalog.🤖 Generated with Claude Code