Skip to content

feat(core): v1 core service with identity, marks, redaction, stats, export and import (AFDEV-13) - #5

Merged
foae merged 3 commits into
mainfrom
feat/AFDEV-13-core-service
Sep 29, 2026
Merged

foae merged 3 commits into
mainfrom
feat/AFDEV-13-core-service

Conversation

@foae

@foae foae commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

The transport-neutral v1 service layer, on top of the decoder (pkg/envelope), the canonical writer (pkg/canonjson) and the v1 store (internal/store). Nothing serves it yet; the HTTP routes land next.

What changed

  • v3 core parked. internal/core moves unchanged to internal/v3/core (import paths only, first commit), so the running v3 service and its e2e gate stay green.
  • Identity. canonjson.Sum (hex SHA-256 of the canonical bytes); (*envelope.Envelope).IdentityTree and ContentHash over the identity members. Every conformance/hash/ vector passes through the decoder and through Service.Create.
  • envelope.DecodeWithLimits. Decode is DecodeWithLimits(body, BodyLimit, MaxDepth); import re-decodes stored envelopes under the 32 MiB import cap (a stored envelope can exceed the 10 MiB create cap after U+FFFD replacement) and two levels of depth headroom (StoredDepthHeadroom: the nesting inference can add).
  • Contract change: trim again after a cut. A member with x-trim that truncation cut is trimmed again (step 5 in conformance/README.md, reference implementation, new fixture trim-after-truncate). Without it a cut could leave trailing whitespace that a second pass removes, and the service could not re-import its own export. FuzzDecode now asserts the decoder is idempotent on its stored output. No existing fixture's expected output changed.
  • internal/core (no net/http in its dependency closure, enforced by a test):
    • Create: keyed replay (200), key reused with different content (409 replay_mismatch, key_reused detail with existing_id), 24-hour keyless window skipping processed rows, UUIDv7 uid; lookups and insert in one BEGIN IMMEDIATE transaction.
    • Get, List (cursors, q, presence-aware filters validated in one place shared with Stats), Stats.
    • Mark / MarkBatch: strict body decoding, updated/unchanged/not_found classified in the same write transaction as one set-based update.
    • Redact: tombstone keeps identity, hash and processing fields; repeat is unchanged.
    • Export: format 2 streamed in one read transaction with header, trailer and digest.
    • Import: format 2 verified in full before the write transaction (strict record parser, count and digest, uid and timestamps, recomputed content_hash; tombstones recognised by redacted_at keep their hash), then uid dedupe and key conflicts reported, never failed.
    • Meta, Schemas, Schema; typed *Problem errors carrying the contract's codes and pointers (?name for parameters); a busy database is 503 unavailable.
    • Records encode stored payload and context bytes through unchanged, member order as the Submission schema, no HTML escaping.
  • docs/develop.md: layout and the frozen-hash rule updated.

Tests

  • Every conformance/decode/ fixture and every conformance/hash/ vector through Service.Create: status and code, warnings as a multiset of (code, pointer), stored envelope as canonical bytes, content_hash.
  • Concurrency: parallel keyed creates with one key, and parallel keyless identical creates, each yield one row.
  • Redaction and marks: tombstone fields, keyed replay of the original returns the tombstone, re-marks keep processed_at, processed=false clears all five.
  • Property test over seeded data: stats totals equal list totals for random filters, plus group and open/processed invariants.
  • Export → import → export over every accepted fixture is byte-equal (including a record over 10 MiB after U+FFFD expansion, a member cut at its limit next to whitespace, and a body at the depth limit whose member inference moves under payload); re-import is a no-op; key conflicts listed; tampered count or digest writes nothing.

Gates

just check, go test -race -count=1 ./..., just fuzz, the e2e suite against the built binary (55/55), shellcheck on both skills, bash tests/skill/run-tests.sh (124/124), just contract, and staticcheck 2026.2.1: all green locally.

Moved unchanged, import paths only, so the running v3 service and its e2e gate stay green while the v1 core takes internal/core.
…xport and import (AFDEV-13)

internal/core is the transport-neutral v1 service on the decoder, the canonical writer and the v1 store: create with keyed replay, 409 on key reuse and the 24-hour keyless window; get and list with cursors, q and presence-aware filters; single and batch processing marks; redaction tombstones; stats; export format 2 streamed with its digest; import format 2 verified before the write transaction, uid preserved, key conflicts reported. pkg/canonjson gains Sum and pkg/envelope the content_hash identity and DecodeWithLimit for the import cap.
Normalisation trims a trimmed member again after a truncation cut (new fixture trim-after-truncate, reference and README updated), so the decoder is idempotent on its stored output; FuzzDecode now asserts that. Import re-decodes with two levels of depth headroom for the nesting inference can add. Record members follow the Submission schema order; a blank verdict, ref or processed_by clears the field; import requires RFC 4122 version and variant bits; requested but empty stats groups and series are present; the missing-member error is deterministic.
@foae
foae merged commit bf95e0a into main Sep 29, 2026
4 checks passed
@foae
foae deleted the feat/AFDEV-13-core-service branch September 29, 2026 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant