Repository navigation
feat(reference): add a resolver for anchors and transclusions - #45
Merged
Merged
Conversation
A case of kind `anchor` holds a small repository: the current snapshot
under tree/, readable earlier blobs under blobs/, and optionally the
three sides of a merge that the runner performs with stock git. It
lists records and referrers with the outcome expected for each: the
status and reason, the anchored leaf, the span, the suggestion gate
with the written document, and every transclusion with its unit.
Authors write `{{blob:PATH}}` where an object id goes; the runner
replaces it with `git hash-object`'s id in a repository of the case's
object format. check_tree.py checks anchor cases for their files,
their controls, and that every asserted field is compared.
No anchor case is added: the cases come from an author who has not
read an implementation.
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
reference/blockspec/anchor.py implements spec section 11 on top of the reader: the record and pin grammars with their canonical forms, correspondence (the leftmost chain, LCS inside gaps, EDITED pairing with integer thresholds), every refusal rule and reason, the span search with the edge rule and the alignment check, the statuses, the suggestion gate and write-back, replies, and transclusion. Where the spec leaves an input open, the resolver reports it as UNDECIDED with the question rather than choose an answer. document.py gains resolve(), the anchor kind's entry point. It loads anchor.py from its own directory by path and hands it this file's reader, so a mutated copy of the reader is the one it runs. The reader's parse output is unchanged. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The reader's resolve() loads anchor.py from the directory it sits in. The gate's scratch copy sat in conformance/, where it would find no anchor.py, and every anchor case would then fail against the unmutated baseline once such cases exist. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
check_tree.py now reports a placeholder target outside tree/, blobs/ and merge/ whose bytes equal a readable file's: ids are content-addressed, so such a file names a readable blob. Alterations now also turn a 0 or 1 index into a bool, and a test kills a not_integers that checks floats alone. The schema now requires a transclusion's leaf to be a paragraph and allows unmappable-offset only on RESOLVED or EDITED, as the README says. The self-test's good body follows that, and pins ten more rules. The README states the landing order of the anchor cases and the resolver, warns about content addressing, notes git's CRLF conflict markers, and lists what a tool writes among what a case cannot express. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
CameronBrooks11
added a commit
that referenced
this pull request
Oct 16, 2026
## What this adds A second case kind, `anchor`, so that the conformance tree can state what `SPEC.md` §11 resolves: comments, suggestions, replies and transclusions, across edits and stock `git merge`s. It adds the runner support, the tree checks and the documentation. **It adds no anchor case**: under the standing rule (org contract §2.1), §11's cases come from an author who has not read an implementation, and they are to be written from `SPEC.md`, 0003 and `conformance/cases/README.md`, *The kind `anchor`*, which this PR adds. ## Landing order The first anchor case turns the required `reference` check red, because the reader has no `resolve` until the resolver (#45). So: after this merges, the anchor cases are written blind from `main`, on a branch and pull request of their own, and reviewed there. #45 is then rebased onto the cases branch, run against them with no case changed, reviewed, and merged. That merge lands the cases and the resolver together, and the cases' pull request closes as landed through #45. `conformance/cases/README.md`'s anchor section states the same. ## The case shape cases/anchor/<group>/<case-name>/ expect.json WHY.md tree/<path> the current snapshot, each file at its path blobs/<any> more readable blobs, such as a record's base merge/{base,ours,theirs}/<path> optional: a merge that makes <path> <anything else> not in the repository (a placeholder's target, a written file) - **Placeholders.** `{{blob:PATH}}` in any file is replaced by `git hash-object`'s id of the case file at `PATH`, in a repository of the case's `object_format` (`sha1` by default, or `sha256`). Authors write content, never hashes. A file outside `tree/`, `blobs/` and `merge/` can be named but is not readable, which is how `base-unavailable` is set up. - **Merges.** `"merge": {"policy.md": "clean" | "conflict"}` runs kindkit's `gitmerge.merge` (stock git, no configuration from the machine) over the three sides; the working-tree result, markers and all, becomes the current file, and git's outcome is asserted. - **`records`**: per record path, `status` (the six statuses of §11.6 with §11.10's reasons, plus `REPLY` / `#REF!(reply-to)` for replies), `leaf` (the paired current leaf as a node path), `span` (code point offsets in that leaf's §11 text), and `suggestion` (`null`, or the §11.7 gate: `apply` with a case file holding the exact written-back document, or `stale` with `null` / `unmappable-offset`). - **`transclusions`**: per referrer, every transclusion paragraph in document order, each with its `leaf`, `status` (`RESOLVED`, `REFUSED(<reason>)`, or `#REF!(<name>)`) and `unit` (the target document's resolved path and what the name binds now). - **`controls`**: required when any outcome is `REFUSED(...)`, as for a `parse` refusal. The implementation is handed `resolve(repo, queries)` with the tree's texts and the readable blobs by id, so it needs no git (`run_cases.py`'s docstring). ## Where it is checked - `just conform` and the mutation gate dispatch `anchor` cases through `run_cases.py`; an implementation without `resolve` fails each one. - `just tree` (`check_tree.py`, stdlib only) checks each anchor case's right answer passes, an empty answer fails, every asserted field is compared (one alteration at a time, an index or offset given as a float or a bool included), its files are consistent (placeholder targets, merge sides, written files, listed records and referrers, and no unreadable placeholder target with a readable file's bytes, since ids are content-addressed), and its controls. - CI's `conformance` job runs `conformance/schema_selftest.py`, which feeds kindkit's own validator one valid synthetic anchor body and 44 broken ones, each in a tree of its own, and requires each broken one to be rejected. The empty-implementation step now defines an empty `resolve` too, so it keeps failing every case once anchor cases exist. `tests/test_anchor_kind.py` drives the runner and `check_tree.py` over synthetic trees in a temporary directory, with a fake implementation that checks the repository it is handed (ids recomputed with `hashlib`, merge results, queries) and echoes a canned answer. Its bodies are not conformance cases and say nothing about how an anchor resolves; flagging them here in case they should go. ## Gates $ just check All checks passed! $ just test 26 passed in 2.97s $ just conform 0 failure(s) across 237 case(s) in the fixture tree $ just tree 237 cases: 160 accept, 77 refuse, 0 anchor 0 problem(s) $ just mutants 91 killed, 0 survived, 0 equivalent, 0 stale, 0 broken $ python3 conformance/schema_selftest.py <kindkit checkout at 6ebd80d> 0 broken body(ies) not rejected $ python3 <kindkit>/tools/validate_case_tree.py conformance/cases 0 invalid case(s) across 237 case(s) in 1 tree(s) ## Break-checks Each new check was broken and seen to fail, by a sweep run locally (not committed) that hashes the file before and after each mutation and reports BROKEN when a mutation does not land. Runner and `check_tree.py` mutants run `pytest tests/test_anchor_kind.py`; schema mutants run the self-test. killed compare skips span killed compare skips status killed transclusion length unchecked killed written text unchecked killed stale reason unchecked killed placeholder not substituted killed object format ignored killed blobs/ not readable killed merge sides not readable killed merge outcome unchecked killed merge sides swapped killed nested placeholder allowed killed merge over a tree file allowed killed merge messages dropped killed check_tree: empty answer not tried killed check_tree: controls not required killed schema: leaf nullable whatever the status killed schema: apply on any status killed schema: unit allowed when refused killed schema: anchor body open killed bools taken for integers killed no bool alterations killed unreadable-blob lint off killed schema: transclusion leaf any type killed schema: unmappable-offset on any status 25 killed, 0 not ## After review Added in a second commit: the content-addressing lint above; bool alterations and a test that kills a `not_integers` checking floats alone; the schema now requires a transclusion's leaf to be a paragraph and allows `unmappable-offset` only on `RESOLVED` or `EDITED`, as the README says (the self-test's good body no longer breaks that); ten more rules pinned in the self-test; and the README now notes git's CRLF conflict markers and lists re-anchoring, anchor creation and re-pinning among what a case cannot express. ## Notes for review - The `parse` kind is unchanged: its schema branches are untouched (only `accept` moved from the top-level `required` into a `parse` branch), and `just conform`, `just tree` and `just mutants` report what they did before. - `SPEC.md` §10 says every case has the kind `parse`. That stays true until the first anchor case lands, which is the change that should update it. - This changes the runner and the case-tree format, so it is load-bearing under the org contract §3 and wants more than one review pass. --------- Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add 141 cases of the anchor kind under conformance/cases/anchor/, written from SPEC.md section 11 and adjudication 0003 with its addendum, without reading any implementation. They cover every resolution status and reason, the record and pin grammars, the correspondence thresholds at their exact boundaries, the leftmost chain, the twin, refill and near-copy rules, context edges, the suggestion gate and write-back, replies, transclusion, merges with stock git, and the residuals R1 to R4. Each WHY.md quotes the passage that decides it, and the README records their provenance. The reference has no resolve yet, so every anchor case fails `just conform`. They land together with the resolver, as the README's landing order says. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
No leaf of base has the refused type with the record's text, so the leaf-at-nth rule gives the same outcome, and the case cannot go red for the T check alone. Its WHY.md now says so. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add seven anchor cases: the 0.15 margin on the base leaf's side, the gap alignment skipping the base element first, a short suffix that must end at the leaf's end, a matching heading slug that is not a transclusion target, a pins line with more after it, a list item after another frontmatter key that is not refused, and pins matched by their raw path. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The second addendum renumbers section 11.9's steps, drops the EXACT rule's gap clause and widens two rows of section 11.10. Re-quote the WHY.md passages that changed. No expected outcome changes. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add 35 anchor cases for 0003's second addendum: an over-long context, the record and transclusion reason orders through cases that set up several reasons, unresolvable-path before broken pins, the line breaks that cannot be written back, notes files that are not ULIDs and malformed reply-to ids, slug-only names, referrers refused as a whole, a refused record body, and the checks moved into step 1 of the order. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Input problems are now reported in section 11.10's order, the first that applies deciding, where the resolver had reported two at once as undecided. A context longer than 32 code points, a notes-directory entry that is not a <ULID>.md record, a reply-to that is not a ULID, and a record whose body the profile refuses are record-inconsistent. A line break in replace that cannot be re-emitted (a list item's first line, a lazy line, a tab in the container prefix, a last line with no ending) is stale as unmappable-offset. The repeat rule drops the clause the addendum removes. Transclusion follows section 11.9's steps 0 to 4: a referrer the profile refuses is refused as a whole, the path precedes the pins, a name only a heading slug matches is #REF!(name) before the pin is read, and the first pinned leaf in document order decides among several pinned reasons. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
An unbound name with an unreadable pin is base-unavailable, since only a slug-only name is decided before the pinned blob is read. A pinned refusal is reported before name-rebound, step 3 before step 4. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Raises run_cases.MIN_CASES to 422, the tree with the anchor cases. The status in README.md and AGENTS.md no longer says 0003 has no cases or implementation: it says the cases were written without reading the resolver, which was written before them and run against them with no case changed. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
just mutants now runs a second table over reference/blockspec/anchor.py, probed by the anchor cases. Its probe writes each mutant as anchor.py beside a copy of the reader, which loads anchor.py from its own directory. Mutants that only turn an undecided input into a guess are left out, as are two that no case field can observe, and two are claimed equivalent with their reasons. The EDITED branch's near-copy call is written near(True) so the gate can name it, since kindkit matches token runs and ignores comments. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add ten anchor cases: U+0080 escaped and raw, a negative nth, two non-canonical reply-to lines, write-back after and across a tab of list-item indentation and after a marker followed by a tab, and line breaks in replace on a list item's lazy line and under a quote's tab. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The case pinned a span that crosses a tab partly consumed by list indentation. That mapping is not decided yet, so the case comes out until it is. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add a case with two twins in the gap of a moved leaf, one refused as repeat and one as unaccounted-copy, and two write-back cases whose span lies in a leaf with a tab-indented line without reaching the tab. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Section 11.8 and 11.10 now render #REF!(reply-to) only once a reply's document checks pass. Re-quote the three WHY.md files that cited the old wording. No expected outcome changes. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add twelve anchor cases: a quote that overlaps itself and overlapping hits, replies whose document is missing or refused with their place in the reason order, spans across, before and after the spaces a partly consumed tab leaves in a code block, including under a fence's own indentation, and a span ending at a code block's final line ending with its control. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
A JSON array where the grammar wants an object raised out of resolve(); objects are now told from arrays, and every malformed record or pin is record-inconsistent. Link text that holds a link is no link (CommonMark example 518), so such a paragraph is no transclusion. An image is no link, prose with no inline link is no transclusion, a title may be followed by a line ending, and an entity reference is decoded only with its semicolon. The self-transclusion walk skipped a whole document when one of its paragraphs was undecided; it now skips only that paragraph. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Drops the claim that leaving MOVED leaves in the gaps is equivalent: with twins beside a move, it swaps repeat and unaccounted-copy. It is now an ordinary mutant. Adds mutants for the parsing fixes the cases can observe. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The 0001 and 0002 cases came before the reader; the 0003 cases came after the resolver, written without reading it. AGENTS.md and SPEC.md said the whole suite came first, and the cases README said the resolver's branch is rebased onto the cases, where it is merged. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
A reply now goes through its document's checks, document-missing and not-in-profile, before it is looked up by its reply-to. A span whose last code point is the line ending of a code block's last content line is stale as unmappable-offset. Overlapping occurrences were already counted by start position, for hits too; a test now pins that. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The tree now holds the final anchor cases: 209 of them, 446 in all. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Adds resolver mutants for a reply's document checks and for a span that ends with a code block's final line ending; the anchor cases kill both. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Add six anchor cases: an anchor that is a JSON array, and referrer paragraphs that are not a transclusion under CommonMark 0.31.2 (a link inside link text, an entity with no semicolon in the title, an image, prose holding the title's characters), plus one whose title is followed by a line ending before the closing parenthesis, which is. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
The tree now holds 215 anchor cases, 452 in all. Signed-off-by: Cameron Brooks <95648474+CameronBrooks11@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reference resolver for
SPEC.md§11, together with §11's conformancecases. The cases are #46's,
test/anchor-casesatd389dee(215 anchorcases, 452 in all), written from
SPEC.mdby an author who had not read theresolver, and merged here with no case changed. Merging this lands both, and
#46 closes as landed through it.
main, with 0003's three addenda (#43, #47,#48), is merged in too.
Landing order (decided). The first anchor case would turn the required
referencecheck red onmain, since the reader had noresolveuntilthis PR. So the cases were written blind from
mainon their own branch andreviewed there, then merged into this branch and run against it.
README.md,AGENTS.mdandSPEC.mdnow say that the 0001 and 0002 casescame before the reader, and the 0003 cases after the resolver.
Against the cases
No case failed, here or at the earlier merges of
e0f5525(422 cases) and7375c2c(446). So nocase is wrong per SPEC, no SPEC question came out of a failure, and the
resolver needed no fix for the cases.
run_cases.MIN_CASESis 452.What this adds
reference/blockspec/anchor.py, standard library only, implements §11 ontop of the existing reader:
(key order, spacing, escape set,
Bas 40 or 64 lower-case hex digits byobject format,
Nas plain decimal),replacetied tokind, ULID filenames and
reply-to, a body the profile refuses, and the consistencyrules against
base, with occurrences counted by start position;alignment inside each gap with MOVED leaves left out, and EDITED pairing
with the integer comparisons of the first addendum (
22/30 − 42/72istested as exactly 0.15);
order;
the alignment check;
NFC-to-raw mapping through container prefixes, lost indentation, CRLF and
bare CR; a
\ninreplacewritten as the line's ending plus itscontainer prefixes; and
unmappable-offsetfor a span that does not map,for a line break that cannot be re-emitted, for a span crossing or ending
right before a tab's leftover spaces, and for a span ending with a code
block's final
\n;to 4, with a referrer the profile refuses reported as
"REFUSED(not-in-profile)"and the first pinned leaf in the pinned blob'sdocument order deciding among several
pinned-<reason>s.document.pygainsresolve(repo, queries), the interface the runnercalls. It loads
anchor.pyfrom its own directory by path and hands it thisfile's reader, so a mutated copy of the reader is the one the resolver runs.
The reader's own output is unchanged: a script run locally compared
conform()fromorigin/mainand from this branch, messages included, forevery
parsecase input, and got237 inputs, 0 differ. A copy with oneline changed gives
237 inputs, 35 differ.Fixed after review
anchor: [1], a leaf ora pin that is an array) raised out of
resolve(). Objects are now toldfrom arrays, and every malformed record or pin is
record-inconsistent.The review's fuzz (6,000 random records, pins and referrers) now raises
nothing.
[a [b](c.md) d](policy.md#scram "transclude")was read as atransclusion. Link text cannot hold a link (CommonMark 0.31.2, example
518), so it is no link and no transclusion.
paragraphs was undecided, so A to B to A went unseen. It now skips only
that paragraph.
twins beside a move, the mutant swaps
repeatandunaccounted-copy.It is an ordinary mutant now, a unit test kills it, and
anchor/repeat/twins-beside-a-movekills it in the gate.UNDECIDED: an image titledtranscludeis no link; a paragraph with no inline link is notransclusion; a title may be followed by a line ending before
); anentity reference is decoded only with its
;, sotranscludeis nottransclude.The mutation gate
just mutantsgates the resolver as well as the reader: a second table inconformance/mutants.pyoverreference/blockspec/anchor.py, probed by thewhole tree. Its probe writes each mutant as
anchor.pybeside a copy ofthe reader. Mutants that only turn an
UNDECIDEDinto a guess are leftout, since no case can expect one, as are the duplicate-id warning and a
notes-directory subfolder, which no case field can observe. The
resolver's table, as the gate printed it:
No survivors. The six holes the last run reported, one per review fix,
are now killed by the cases #46 added for them.
One equivalence claim, "item first line repeated": a list item's first
line holds its marker in the prefix region, so the lazy-line check refuses a
break there anyway. kindkit flags a claim only when the suite kills the
mutant; it cannot show that one the suite does not kill is equivalent, so
this rests on the argument.
Where SPEC.md is still open: refused, not guessed
Each of these returns
{"status": "UNDECIDED", "question": ...}for thatrecord or transclusion, which no case can expect.
Left open by §12:
title quoting form or a title with backslash escapes or entity references
(one that decodes to
transcludebut is not written"transclude"), atranscludelink in a paragraph that holds other text, a transclusion ina list item or block quote, no pin, and a document that transcludes
itself, directly or through another.
(read here as not ending in
.md).Not in §12, and not decided either:
#name.link text ends under CommonMark.
prefixed differently: §11.7 writes each
\nas the ending of "the lineit breaks", and such a span has two candidates.
segment and self-transclusion after the path resolves, the non-markdown
target before the pins, and no pin at step 1.6.
tree, and a record path in no notes directory.
Cheap to decide but left
UNDECIDED(CommonMark settles them, at thecost of a full inline parser): prose that holds
"transclude")alongside areal inline link elsewhere in the same paragraph.
Readings taken, not refused, worth confirming:
line's prefixes, so
> a\n> bwitha\nreplaced byxgives> xb."transclude")in a link-like shape is treated as apossible transclusion and refused as undecided where it is not exactly one
link. That refuses more than needed, and never resolves wrongly.
Gates
Break-checks of the unit tests
tests/test_anchor.pywas break-checked by a sweep run locally (notcommitted). It hashes the file before and after each mutation, reports
BROKEN when one does not land, runs the tests, and restores the file. The
latest run gave
105 killed, 1 not, the one being the equivalent mutantabove. The hook and the gate's scratch location:
3 killed, 0 not.Not covered
measures them on real history (0003 §12).
implemented: the format does not require them.