Skip to content

feat(envelope): pkg/envelope decoder and pkg/canonjson writer (AFDEV-11) - #3

Merged
foae merged 2 commits into
mainfrom
feat/AFDEV-11-envelope-decoder
Sep 28, 2026
Merged

foae merged 2 commits into
mainfrom
feat/AFDEV-11-envelope-decoder

Conversation

@foae

@foae foae commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

What

Package 3 of the v1 write path: pkg/envelope, the decoder, plus pkg/canonjson, the contract's canonical JSON writer.

  • pkg/envelope: Decode(body) (*Envelope, []schema.Detail, error). A token-stream parser (RFC 8259 only; number spellings kept as json.Number; duplicate members last-wins with one duplicate_key and the discarded value's warnings dropped; invalid UTF-8 repaired one U+FFFD per maximal subpart; lone surrogate escapes replaced; 512-level nesting limit), then the inference table in the contract's order, the normalisation order via pkg/schema, the kind schema as a guide for an explicit kind, and the envelope's recommended check. Only the four bodies the contract names are refused, as a *Rejection{Status, Code, Message}. Envelope.Tree() is the JSON tree identity and the store will consume.
  • Warnings in a path trie during parse and inference: duplicate drop, remap and collapse are subtree operations, so a 10 MiB body with a warning per member decodes in linear time (a literal port of the reference's slice rewrites is quadratic; a test pins this).
  • Limits from the compiled envelope schema at init (byte limits, trim/token, truncation, context entry and value limits, recommended list, member set); body limit, depth, schema_version rule and maximum are constants checked against conformance/ by tests.
  • pkg/canonjson: Marshal, Append, AppendString on the write path's JSON tree; panics on any other Go value, a malformed json.Number or invalid UTF-8. internal/canonjson stays as the frozen v3 event-hash form (develop.md now says which is which).

Contract changes

  • The inference table now states that unknown members move into payload in code point order of their names (observable only when one of them is named moved). The reference used dict order and crashed with a KeyError on a top-level moved with no payload.moved; the context-overflow pointer remap also ran before placement, so under a wrapped payload.moved the pointers were wrong. Both fixed in conformance/reference/decode.py; new fixtures moved-named-member-order and context-overflow-under-wrapped-moved pin them. All 87 fixtures agree with the reference and with the Go decoder.

Tests

  • One named test per inference-table row reading its fixture (with a row-specific assertion), every interaction fixture, the hash fixtures' bodies projected onto the identity members; the fixture directory sets must equal manifest.json and the row table.
  • Parser: grammar rejections, spellings, escapes, depth, duplicates, pointer escaping; UTF-8 repair vectors from Unicode table 3-8 plus an exhaustive cross-check of every 1- and 2-byte sequence and shaped 3/4-byte sequences against an independent oracle formulation.
  • FuzzDecode: seeded with every fixture body plus hand-written cases under testdata/fuzz/FuzzDecode; oracle json.Valid + token scan for object/depth; invariants on the stored shape and a canonical fixed point. just fuzz (30 s) is a new CI step and gate.

Gates

All green locally: just check with a clean tree, go test -race -count=1 ./..., just fuzz, live e2e 55/55, shellcheck (both forms), skill tests 124/124, just contract (87 fixtures), staticcheck 2026.2.1 clean.

Out of scope: identity and hashing (AFDEV-13), storage (AFDEV-12). The running service still implements the v3 API; nothing here is reachable from it yet.

Review round

Panel default on 24a6894 (4 of 5 ran). Fixed in the follow-up commit: warnings of a value discarded by an inference replacement (payload.moved.<name>, context.occurred_at_raw) are dropped, in Go and in the reference, with two fixtures; pkg/canonjson validates member names; the fuzz fixed-point invariant runs through the raw parser instead of Decode; the invalid-escape rejection message no longer echoes a body byte. Acknowledged: warning pointers are unbounded because member names are, so a small body can produce a very large warning list; that is the contract-level question already tracked as a Backlog follow-up, not something this package can decide.

pkg/envelope is the v1 decoder: a token-stream JSON reader that keeps number
spellings, resolves duplicate members (last wins, warnings of the discarded
value dropped), repairs invalid UTF-8 one U+FFFD per maximal subpart and
replaces lone surrogate escapes; then the inference table in the contract's
order, the normalisation order through pkg/schema, the kind schema as a
guide and the envelope's recommended check. Decode rejects only the four
bodies the contract names (over 10 MiB, not JSON, not an object, nested
deeper than 512) as a Rejection with the status and code; every other JSON
object is stored. Warnings live in a path trie so that dropping, remapping
and collapsing them stay subtree operations on a 10 MiB body.

pkg/canonjson is the canonical JSON writer of the contract (members in
code point order, numbers verbatim, minimal escaping), needed at decode
time for the coerce rows and by the fixture tests; identity hashing builds
on it later. internal/canonjson stays as the frozen v3 event-hash form.

Tests: one named test per inference-table row reading its fixture, every
interaction and hash fixture, directory sets checked against manifest.json,
parser and UTF-8 vectors with an independent oracle, a linear-time check on
warning-heavy bodies, and FuzzDecode seeded with every fixture body
(just fuzz, 30 s, wired into CI).

Reference: unknown members now move in code point order of their names
(README row updated); _place no longer confuses a member named "moved"
with placement under payload.moved (a KeyError before); the context
overflow remap runs after placement so a wrapped payload.moved does not
catch it. Two interaction fixtures pin those corners.
…writer key check, fuzz fixed point (AFDEV-11)

A moved member replacing a value under payload.moved.<name>, and an
unparseable occurred_at replacing a producer's context.occurred_at_raw, now
drop the discarded value's warnings before duplicate_key, as the parser
does for a duplicate member; the reference did the same and kept them,
leaving pointers into content the record no longer holds. Two fixtures pin
both paths; the README rows say so.

pkg/canonjson checks member names for valid UTF-8 like it checks values.

FuzzDecode's fixed-point invariant goes through the raw parser and the
writer, not Decode: a stored envelope is not always an admissible request
(nothing is re-trimmed after a cut). The rejection message for an invalid
escape no longer echoes the body byte; the did-you-mean hint uses the
contract's simple case mapping.
@foae
foae merged commit 18a9f43 into main Sep 28, 2026
4 checks passed
@foae
foae deleted the feat/AFDEV-11-envelope-decoder branch September 28, 2026 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant