docs(matrix): audit the feature matrix against what actually ships - #1196
Merged
Merged
Conversation
The matrix claimed two shipped document types were unavailable, which is the
failure its own "Keeping this honest" note warns about -- a row claiming a gap
that has since been filled sends people to Studio Pro for work mxcli can do.
- Microflow rules sat under "Not Yet Implemented" ("no MDL surface at all")
while CREATE [OR MODIFY] RULE ships with a skill, rules.mdl, a
`microflow.rule` syntax topic, a CATALOG.RULES view and rule REFS edges.
- Message definitions likewise: SHOW / DESCRIBE / CREATE OR MODIFY / DROP /
ALTER all ship, with tests and examples.
Both now have rows in Core Document Types instead.
Layouts were recorded as read-only in four places -- the row's CREATE, OR
MODIFY, DROP, ALTER, Examples, Tests, Skills and Syntax cells were all N, and
three gap lists repeated "Read-only". All eight are Y. OR MODIFY was verified
at exec rather than parse: `create or replace` reports Created then Unchanged,
and `create or modify` over the stored layout reports Unchanged.
Sixteen Examples cells corrected against the doctype-tests listing, including
two that cited the wrong file -- Task Queues and Scheduled Events both pointed
at 21 (import/export mappings) and the mapping rows pointed at 06 (the REST
client).
Rule counts are measured, not remembered: 19 built-in (`lint --list-rules`
with no project rules dir) and 31 Starlark (`.claude/lint-rules/*.star`). Four
different wrong pairs were in circulation across five live docs, one claiming
41 built-in rules. CHANGELOG and the dated eval proposal are left alone: they
are records of a past state, not claims about the present.
The guard could not have caught any of it. TestCapabilityDocsDoNotClaim...
reads only the "Not Yet Implemented" section and checks a hand-maintained list
of seven capabilities naming neither rules, message definitions nor layouts.
So the list is hoisted to `shippedCapabilities`, extended with all three, and a
second test asserts that "Missing Syntax Topics" may not list a capability
whose `mxcli syntax` topic resolves -- a flat contradiction, which is what
makes it mechanically assertable. It is deliberately not extended to the
Skills and Examples gap lists, where an entry can be true at the same time as
a syntax topic exists (Regular Expressions has a topic and no skill).
Checked by reinstating all three false claims: two tests fail, naming each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AI Code ReviewSummaryThis PR audits and corrects the MDL feature matrix documentation against what actually ships in the codebase, plus adds guard tests to prevent future documentation drift. It fixes multiple instances where the matrix incorrectly claimed features were unavailable or had limited support. Critical IssuesNone found. Moderate IssuesNone found. Minor Issues
What Looks Good
RecommendationApprove - This PR successfully addresses documentation drift with both immediate corrections and preventive measures. The changes are minimal, focused, and improve the reliability of the documentation. The added tests provide lasting value by preventing similar drift in the future. No changes are needed. Automated review via OpenRouter (Nemotron Super 120B) — workflow source |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An audit of
docs/01-project/MDL_FEATURE_MATRIX.mdagainst what actually ships, plus the guard extension that would have caught the drift. Every cell changed here was measured against the repo, not inferred.Two shipped document types were listed as unavailable
This is the failure the matrix's own "Keeping this honest" note warns about — a row claiming a gap that has since been filled is worse than no table, because it sends people to Studio Pro for work mxcli can do.
CREATE [OR MODIFY] RULEships, with a skill,rules.mdl, amicroflow.rulesyntax topic, aCATALOG.RULESview and rule REFS edgesSHOW/DESCRIBE/CREATE OR MODIFY/DROP/ALTERall ship, with tests and40-message-definition-examples.mdlBoth now have rows in Core Document Types.
Layouts were recorded as read-only in four places
The row had
CREATE / OR MODIFY / DROP / ALTER / Examples / Tests / Skills / Syntaxall N, and three gap lists repeated "Layouts — Read-only". All eight are Y.OR MODIFYwas verified at exec, not parse — parsing proves nothing about upsert semantics:Sixteen
Examplescells, two of them pointing at the wrong fileCorrected against the actual
mdl-examples/doctype-tests/listing. Task Queues and Scheduled Events both cited21, which is21-import-export-mapping-examples.mdl; the mapping rows cited06, which is the REST client.Rule counts: measured, not remembered
19 built-in (
lint --list-ruleswith no project rules dir) and 31 Starlark (.claude/lint-rules/*.star). Four different wrong pairs were in circulation across five live docs — one claiming 41 built-in rules.CHANGELOG.mdand the dated eval proposal are left alone: they are records of a past state, not claims about the present.The guard could not have caught any of this
TestCapabilityDocsDoNotClaimShippedFeaturesAreMissingreads only the "Not Yet Implemented" section, and checks a hand-maintained list of seven capabilities naming neither rules, message definitions nor layouts. So:shippedCapabilitiesand extended with all three;mxcli syntaxtopic resolves — a flat contradiction, which is what makes it mechanically assertable.Deliberately not extended to the Skills and Examples gap lists: an entry there can be true at the same time as a syntax topic exists (Regular Expressions has a topic and genuinely has no skill), so that check would false-positive.
Verified by reinstating all three false claims — two tests fail, naming each one. A guard only ever run against green docs has not been shown to detect anything.
Testing
make testexit 0, zero failures.make lint-goclean.make check-wiki-pagesclean.One trap worth recording for the next person:
mxcli syntax <unknown-topic>exits 0 while printingUnknown topic, so an exit-code check reports every topic as present. Read the output. That false positive is what nearly made me recorddatasetandxml-schemaas shipping.🤖 Generated with Claude Code