Skip to content

feat: count, filter and format records client-side - #2

Open
luandro wants to merge 1 commit into
mainfrom
feat/cli-ux-filters
Open

luandro wants to merge 1 commit into
mainfrom
feat/cli-ux-filters

Conversation

@luandro

@luandro luandro commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Why

The list commands printed the full JSON body of every record and took no options beyond the project id. Answering "how many observations do we have?" meant piping ~135 KB through jq by hand, and the answer was wrong anyway: the API returns tombstones mixed in with live records, so a project that reported 72 records actually had 56 observations.

What

A client-side query layer — filter → sort → paginate → group → render — applied to observations, projects and alerts.

comapeo-cloud list-observations -p <id> --count                        # 56
comapeo-cloud list-observations -p <id> --count --group-by category
comapeo-cloud list-observations -p <id> --summary
comapeo-cloud list-observations -p "Expedição 2026" --tag tipo-de-corpo-dagua=rio --has audio
  • --count prints a bare number (composes in scripts); --group-by splits it by category, preset, day, device, media or tag:<field>.
  • --summary reports totals, date range, device and media counts.
  • Filters: --category, --tag key=value (v1|v2 for OR, arrays matched), --preset, --since/--until, --bbox, --has, --missing, --search, --sort, --reverse, --limit/--offset, --columns.
  • --format table|json|ndjson|csv, defaulting to table on a TTY and json when piped — so existing | jq pipelines keep working with no flags.
  • -p accepts a project name as well as a 52-character id; ambiguous names list the candidates and exit 2.

All filtering is client-side because /projects/:id/observations accepts no query parameters (verified against a live server and against addDatatypeGetter in digidem/comapeo-cloud). The payload is small enough that this is the honest implementation.

Fixes

  • export-geojson dropped every tag. The properties interface declared categoryId/name/notes, but the emitted object never set them, so every export lost its categories and preset field data.
  • export-geojson wrote to a fixed tmp_export/ in the cwd and leaked it on failure → mkdtemp + finally.
  • -o out.geojson was accepted but produced a ZIP → the extension now decides: .zip bundles media, .geojson is metadata-only.
  • Audio attachments were saved into images/ with a .jpg extension → now audio/ with .mp3.
  • --version printed a hardcoded 1.0.0 while package.json said 1.0.3 → read from package.json.
  • get-attachment --variant <anything> was passed straight through → validated.
  • healthcheck printed an empty JSON body → prints ok.
  • Debug console.log("processing", att) noise removed; downloads are capped at 6 concurrent instead of a 112-wide Promise.all.

Breaking changes

  1. Deleted records are excluded by default. Counts now mean live observations. --include-deleted restores the old view; --only-deleted shows tombstones only.
  2. Default output is a table on a TTY. Piped output is still JSON, so scripts are unaffected.
  3. healthcheck prints ok instead of an empty body.

Version bumped to 1.1.0.

Structure

src/index.ts was 462 lines with every command inline. Split into src/commands/* and src/lib/*, with filters.ts and group.ts pure and unit-testable — no network needed to test the query layer.

npm test now points at bun test, which is what CI already ran: chalk 5 is ESM-only and does not load in jest's CJS runtime, so the new unit tests could not run under jest. 77 tests pass under bun.

Testing

  • 77 unit/integration tests: filter predicates, grouping maths, renderers, project-name matching, env/flag precedence, error mapping, and CLI smoke tests for the new flags and exit codes.
  • Fixture data is synthetic (src/__tests__/fixtures/observations.ts). No real field observations are committed — the projects this CLI targets hold sensitive survey data.
  • Manually exercised against a live CoMapeo Cloud server: --count (56), --count --include-deleted (72), per-category counts, --has audio (21), bbox and date-range filters, --summary (112 photos / 21 audio / 3 devices), plain and zipped exports (tags present in properties, audio in audio/), get-attachment --variant thumbnail producing a valid JPEG, list-alerts, list-projects --count, and JSON/ndjson/csv/table formats.

Not included

Presets can only be grouped by docId prefix (preset:abb1b841): the deployed servers return 404 for /presets, so there is no preset name to resolve. Tracks and fields are 404 on those servers too, so no commands were added for them.

The list commands dumped the full JSON body of every record and accepted
no options beyond the project id, so answering "how many observations do
we have" meant piping 135 KB through jq by hand.

Add a client-side query layer (filter -> sort -> paginate -> group ->
render) and apply it to observations, projects and alerts:

- --count prints a bare number; --group-by splits it by category, preset,
  day, device, media or any tag field
- --summary prints totals, date range, devices and media counts
- filters: --category, --tag key=value (OR with |), --preset, --since,
  --until, --bbox, --has, --missing, --search, --sort, --reverse,
  --limit/--offset, --columns
- --format table|json|ndjson|csv, defaulting to table on a TTY and json
  when piped so existing `| jq` pipelines are unaffected
- -p accepts a project name as well as a 52-character id, and reports
  ambiguous names instead of guessing

Fixes found along the way:

- export-geojson dropped every tag: the properties interface declared
  categoryId/name/notes but the object never set them, so exports lost
  all categories and preset fields
- export-geojson wrote to a fixed tmp_export/ in the cwd and leaked it on
  failure; now mkdtemp + finally
- export-geojson accepted -o out.geojson but wrote a ZIP; the extension
  now decides (zip bundles media, geojson is metadata only)
- audio attachments were written into images/ with a .jpg extension; they
  now go to audio/ as .mp3
- --version was hardcoded 1.0.0 while package.json said 1.0.3
- get-attachment passed an arbitrary --variant straight through
- healthcheck printed an empty JSON body; it now prints "ok"

Deleted records are excluded by default. The API returns tombstones mixed
in with live records (includeDeleted: true server-side), which made counts
misleading: 72 returned records were really 56 observations. Use
--include-deleted / --only-deleted to see them.

Error handling maps 401/403, 404, timeouts and refused connections to
actionable messages, sets exit code 2 for usage errors and 3 for not
found, and no longer calls process.exit() mid-write (which truncated
piped output).

Split src/index.ts (462 lines, every command inline) into commands/ and
lib/, with filters.ts pure and unit-tested. Tests cover the predicates,
grouping maths and renderers against a synthetic fixture; no real field
data is committed. 77 tests, run under bun (chalk 5 is ESM-only, so
`npm test` now points at `bun test`).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant