Repository navigation
feat(envelope): pkg/envelope decoder and pkg/canonjson writer (AFDEV-11) - #3
Merged
Merged
Conversation
pkg/envelope is the v1 decoder: a token-stream JSON reader that keeps number spellings, resolves duplicate members (last wins, warnings of the discarded value dropped), repairs invalid UTF-8 one U+FFFD per maximal subpart and replaces lone surrogate escapes; then the inference table in the contract's order, the normalisation order through pkg/schema, the kind schema as a guide and the envelope's recommended check. Decode rejects only the four bodies the contract names (over 10 MiB, not JSON, not an object, nested deeper than 512) as a Rejection with the status and code; every other JSON object is stored. Warnings live in a path trie so that dropping, remapping and collapsing them stay subtree operations on a 10 MiB body. pkg/canonjson is the canonical JSON writer of the contract (members in code point order, numbers verbatim, minimal escaping), needed at decode time for the coerce rows and by the fixture tests; identity hashing builds on it later. internal/canonjson stays as the frozen v3 event-hash form. Tests: one named test per inference-table row reading its fixture, every interaction and hash fixture, directory sets checked against manifest.json, parser and UTF-8 vectors with an independent oracle, a linear-time check on warning-heavy bodies, and FuzzDecode seeded with every fixture body (just fuzz, 30 s, wired into CI). Reference: unknown members now move in code point order of their names (README row updated); _place no longer confuses a member named "moved" with placement under payload.moved (a KeyError before); the context overflow remap runs after placement so a wrapped payload.moved does not catch it. Two interaction fixtures pin those corners.
…writer key check, fuzz fixed point (AFDEV-11) A moved member replacing a value under payload.moved.<name>, and an unparseable occurred_at replacing a producer's context.occurred_at_raw, now drop the discarded value's warnings before duplicate_key, as the parser does for a duplicate member; the reference did the same and kept them, leaving pointers into content the record no longer holds. Two fixtures pin both paths; the README rows say so. pkg/canonjson checks member names for valid UTF-8 like it checks values. FuzzDecode's fixed-point invariant goes through the raw parser and the writer, not Decode: a stored envelope is not always an admissible request (nothing is re-trimmed after a cut). The rejection message for an invalid escape no longer echoes the body byte; the did-you-mean hint uses the contract's simple case mapping.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Package 3 of the v1 write path:
pkg/envelope, the decoder, pluspkg/canonjson, the contract's canonical JSON writer.pkg/envelope:Decode(body) (*Envelope, []schema.Detail, error). A token-stream parser (RFC 8259 only; number spellings kept asjson.Number; duplicate members last-wins with oneduplicate_keyand the discarded value's warnings dropped; invalid UTF-8 repaired one U+FFFD per maximal subpart; lone surrogate escapes replaced; 512-level nesting limit), then the inference table in the contract's order, the normalisation order viapkg/schema, the kind schema as a guide for an explicit kind, and the envelope's recommended check. Only the four bodies the contract names are refused, as a*Rejection{Status, Code, Message}.Envelope.Tree()is the JSON tree identity and the store will consume.schema_versionrule and maximum are constants checked againstconformance/by tests.pkg/canonjson:Marshal,Append,AppendStringon the write path's JSON tree; panics on any other Go value, a malformedjson.Numberor invalid UTF-8.internal/canonjsonstays as the frozen v3 event-hash form (develop.md now says which is which).Contract changes
payloadin code point order of their names (observable only when one of them is namedmoved). The reference used dict order and crashed with aKeyErroron a top-levelmovedwith nopayload.moved; the context-overflow pointer remap also ran before placement, so under a wrappedpayload.movedthe pointers were wrong. Both fixed inconformance/reference/decode.py; new fixturesmoved-named-member-orderandcontext-overflow-under-wrapped-movedpin them. All 87 fixtures agree with the reference and with the Go decoder.Tests
manifest.jsonand the row table.FuzzDecode: seeded with every fixture body plus hand-written cases undertestdata/fuzz/FuzzDecode; oraclejson.Valid+ token scan for object/depth; invariants on the stored shape and a canonical fixed point.just fuzz(30 s) is a new CI step and gate.Gates
All green locally:
just checkwith a clean tree,go test -race -count=1 ./...,just fuzz, live e2e 55/55, shellcheck (both forms), skill tests 124/124,just contract(87 fixtures),staticcheck 2026.2.1clean.Out of scope: identity and hashing (AFDEV-13), storage (AFDEV-12). The running service still implements the v3 API; nothing here is reachable from it yet.
Review round
Panel
defaulton24a6894(4 of 5 ran). Fixed in the follow-up commit: warnings of a value discarded by an inference replacement (payload.moved.<name>,context.occurred_at_raw) are dropped, in Go and in the reference, with two fixtures;pkg/canonjsonvalidates member names; the fuzz fixed-point invariant runs through the raw parser instead ofDecode; the invalid-escape rejection message no longer echoes a body byte. Acknowledged: warning pointers are unbounded because member names are, so a small body can produce a very large warning list; that is the contract-level question already tracked as a Backlog follow-up, not something this package can decide.