Exposure maps: npoint (pointing squares) and ntaken beside nexp

[cailmdaley]

Claude (Opus) on behalf of Cail.

Each campaign now publishes four exposure-count maps instead of two, all at the mask ladder's nside 131072:

map	counts, per sky pixel, the exposures…
npoint_<run>.hsp (new)	whose pointing square contains it: the convex hull of CCDs 0–35, so chip gaps are filled and the four ear CCDs 36–39 are left out
ntaken_<run>.hsp (new)	with any CCD's imaging area (DATASEC, all 40 CCDs) on it, whether or not the PSF validated
nexp_<run>.hsp	with a PSF-validated CCD on it (unchanged)
nflagged_<run>.hsp	flagged CCD pixels of those validated CCDs (unchanged)

Why. nexp follows the real CCD tiling, so with chip gaps, ears and dithers it comes out as a busy pattern, and the counts pile up where two exposures' ears overlap. npoint is a deliberately smooth, somewhat arbitrary pointing count: every exposure is one filled square. It is the v2 counterpart of v1.X's npoint3 >= 3 cut. sp_validation's mask_v2.0.yaml says v2 has no source for that cut yet, and this map provides one. ntaken alongside nexp gives taken vs successful: ntaken − nexp is the coverage lost to CCDs whose PSF failed validation.

On 44 smk-b05 exposures, npoint is 3 almost everywhere, with seams where neighbouring squares overlap. ntaken and nexp carry the CCD pattern. ntaken − nexp picks out exactly the five CCD footprints that failed PSF validation in that region.

What changed

• exp_maps.py. The per-exposure fragment is now a packed uint16. Bits 0–2 are coverage (1 pointing square, 2 CCD imaging area, 4 validated CCD) and bits 3–10 are the flagged-pixel count, clipped at 255. A HealSparse record-array map was the obvious alternative, but HealSparse stores record arrays uncompressed (33 MB per exposure). Packed fragments are 2.55 MB median, against 2.13 MB for the uint8 fragments they replace.
• Two input sources, one rasteriser. The headers and flag arrays can come from the split scratch store (--exp-dir, what the rule uses) or from the delivered MEFs (--image-mef/--flag-mef, HDU = CCD + 1). split_exp keeps the delivered header's WCS, so the two agree. On exposure 2086785 they gave identical fragments over all 5,373,752 pixels.
• merge_exposure_maps.py sums the fragments into the four maps. It fails, naming the exposure and the backfill command, if it meets an old-format fragment, and it fails if a count would overflow its dtype.
• Wiring. Rule outputs, exposure_maps_targets, README, astra.yaml (new decision masking.pointing_square, and nexp_map_valid_psf_ccds extended to cover ntaken), universes/committed.yaml, and the params pin (only the exposure_maps shell digest moved).

Backfilling existing campaigns

Fragments that existing campaigns already wrote are in the old format, and their scratch stores have been reclaimed. Rerunning exp_maps.py in raw-MEF mode on the staged exposures rewrites the fragment and its exp_maps.json, and snakemake then reads the fragment through durable_edge as before. This takes ~14 s per exposure. On Nibi that is about 5k fragments, so one small job.

Verification

• Not-slow suite: 967 passed, 4 skipped. tests/unit/test_exp_maps.py, test_decisions.py and tests/workflow: all pass. astra-tools validate: pass.
• Unit fixture: a CCD grid with gaps, an ear and one unvalidated CCD. The tests check that the gaps count in npoint and not ntaken, the ear in ntaken and not npoint, the unvalidated CCD in ntaken and not nexp, that split and MEF sources give identical fragments, and that the merge rejects an old fragment.
• Real data: all 44 smk-b05 exposures were rebuilt from raw MEFs. nexp and nflagged are bit-identical to production over the whole campaign footprint.
• Independent review (GPT-6.1 Sol): nothing blocking. The pointing polygon matches analytic tangent-plane coverage across RA = 0/360, at Dec 80° and 89.8°, at both poles and in both WCS handednesses. A reclaimed-exposure DAG probe schedules no exp_maps rebuilds and targets all four maps.

#960 (comment sweep) touches the same two scripts and will need a small merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FD35GJ3MDN7ceDhFwxzMbU

[cailmdaley]

Claude (Opus) on behalf of Cail.

The production backfill (2,550 exposures, b01–b50 on Nibi) turned up a failure on 2 exposures (2105593 and 2869649). On some exposures the CCD corners along one side of the array are collinear to within ~1e-6 rad, and hpg.query_polygon rejects such a hull vertex as a degenerate corner. Fixed in e1ba1d0: straighten() drops hull vertices that turn by less than 1e-4 rad before rasterising. That moves the boundary by < 0.1″, against 1.6″ sky pixels. It has a unit test, and both exposures now build. The other 46 campaigns' rebuilt nexp/nflagged were bit-identical to production.

[cailmdaley]

The review's HOLD identified a destructive rerun path in my automatic raw-MEF recovery design.
Fix-forward commit 313829f adopts the requested lifecycle boundary and removes that automatic recovery path.

• While the store is live, exp_maps produces the declared fragment, and reclamation requires it first.
• After reclamation, existing fragments are immutable DAG leaves.
  A rule-level wildcard constraint removes their rasterization producer, rather than just cutting its input edges.
  Changes to the rasterizer, shell or params therefore cannot schedule a job that deletes those fragments.
• Missing reclaimed fragments fail planning with an explicit out-of-DAG backfill command, even when raw MEFs are still staged.
  The standalone raw-MEF script mode remains available for deliberate backfills.
• The merge's hard failure, exposure list and backfill commands remain unchanged.

Allocation tests in the requested SIF: 38 workflow tests and 8 map unit tests pass.
The six new immutable-fragment cases cover script/shell/params edits, absent raw files, and both present and missing completion manifests: zero exp_maps jobs, no matching producer, unchanged fragment bytes and mtime.
All six fail against the unsafe parent.
The retained resume tests now assert planning failure after reclamation; a live-store regression still verifies scheduling before both merge and reclamation.

The fix and corrected #964 documentation are cherry-picked and pushed to integration/smk-g19 as b4d1f76 and 661bcdf.
Its SIF allocation suite passes: 63 workflow tests and 8 map unit tests, with both normal and noisefill params pins refreshed.

Production remains untouched.
A deliberate backfill is now required for a missing reclaimed fragment; relaunching is not permission to regenerate immutable products.

— Codex on behalf of Cail.