Skip to content

RFC 0001: metadata resolution layer (FORGE-1) - #69

Open
Swimburger wants to merge 21 commits into
cloudflare:mainfrom
Swimburger:rfc-0001-metadata-resolution-layer
Open

Swimburger wants to merge 21 commits into
cloudflare:mainfrom
Swimburger:rfc-0001-metadata-resolution-layer

Conversation

@Swimburger

@Swimburger Swimburger commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Implements RFC 0001 for FORGE-1: a metadata resolution layer between the vendor extensions in a spec and the engine.

What changes

  • A native x-forge-* vocabulary (src/metadata/), listed in full below. Every native key begins with x-forge-; x-sensitive keeps working as a deprecated name of x-forge-sensitive.
  • A compatibility layer (src/compat/, behind the default-on compat feature): the only code that knows x-fern-* and x-stainless-skip. It maps them onto the native keys, and is designed to be deleted as a whole.
  • Native keys win. When a spec writes both, the native value is used whatever the key order; only the winning value is validated, and errors name the key the spec actually wrote.
  • x-forge-group-name takes a dotted string ("accounts.members") or an array, read exactly as groups always have been. The dotted string is Cloudflare's convention rather than Fern's; FORGE-12 tracks moving specs off it and then making the Fern adapter match Fern.
  • A boundary test (tests/boundary.rs) fails if x-fern or x-stainless appears in src/, tests/ or packages/forge/ outside the compatibility layer, doc comments included.

The native vocabulary

All decided native names, with the vendor key the compatibility layer reads in its place:

Native key On Value Compat equivalent
x-forge-method-name operation, alias string x-fern-sdk-method-name
x-forge-group-name operation, alias dotted string ("accounts.members") or array of strings, top level first x-fern-sdk-group-name
x-forge-skip operation, alias (no method) · property schema (no field) boolean x-fern-ignore (operation) · x-stainless-skip (property; a list of targets counts as false)
x-forge-availability operation, alias alpha | beta | preview | generally-available | deprecated x-fern-availability
x-forge-model-name schema in components.schemas / $defs string x-fern-type-name
x-forge-property-name property schema string x-fern-property-name
x-forge-parameter-name parameter string x-fern-parameter-name
x-forge-sensitive schema boolean none (replaces x-sensitive)
x-sensitive schema boolean none: deprecated native name of x-forge-sensitive, read only where that is not written

Existing native keys, unchanged: x-forge-hidden, x-forge-require-confirmation, x-forge-aliases (operation) and x-forge-commands (document root).

Not read, and unchanged: x-fern-enum, x-fern-audiences, x-stability and the other vendor keys without a Forge primitive (FORGE-5). Later renames (group → namespace, the availability word) are FORGE-3.

Compatibility

Not breaking. Existing specs build the same Forge, the WIT changes only in doc comments (forge:plugin@0.3.0 is unchanged), and generated output is byte-identical. Built without compat, the native keys behave the same, and vendor keys become ordinary extensions passed to plugins.

Verification

Checked against the unmodified engine on the released aggregate (openapi.forge.json, 27.6 MB), as detailed in the RFC's Verification section:

  • the original Fern-keyed parse suite passed before any fixture was migrated;
  • forge --target ts writes the same 3,381 files, byte for byte, and the full model dump is identical;
  • native vs WASM output and dumps are identical; wit_is_current holds;
  • cargo test passes with compat on and, run locally, with compat off (CI compiles but deliberately does not test the compat-off build; see the RFC); clippy is clean in the library-only, compat-off and all-features configurations;
  • parse time is within about 1% of main (paired runs of the parse bench), with the same peak memory and allocation count.

Reviewing

The RFC is the place to start, then src/metadata/mod.rs, src/compat/ and src/link/resolve.rs. Most of the diff under tests/parse/ is fixtures moving to native keys.

Open questions for reviewers, also in the RFC:

  • leaving x-fern-audiences unread, which differs from the ticket text;
  • the empty group path: "" and [] should mean the root, but the model has no root methods yet (Known gaps).

Follow-ups: FORGE-3 (the breaking vocabulary renames), FORGE-4 (the versioning RFC), FORGE-5 (a Forge CI lint for vendor keys forge doesn't read) and FORGE-12.

…k (FORGE-1)

Add the metadata resolution layer of RFC 0001.

- src/metadata: the native x-forge-* vocabulary, per kind of object, as
  pure key tables, primitives and value-format descriptors.
- src/compat: the compatibility layer, behind the default-on `compat`
  feature. The Fern and Stainless adapters map their keys onto the same
  primitives; every vendor name lives here.
- src/link/resolve.rs: native first, compat second. A native key's value
  is kept over a compat one's, whatever order they are written in.
- Errors name the key the document wrote.
- Docs and the WIT speak the native keys; the WIT copies are regenerated
  (doc comments only, so forge:plugin stays at 0.3.0).

Output is unchanged: the unmodified Fern-keyed suite passes against it, and
the real aggregate builds the same files and the same model dump.
x-fern-* and x-stainless-skip become their x-forge-* equivalents; a dotted
group string becomes an array. What only the compat layer does is covered
by tests/compat.
tests/boundary.rs fails, naming the file and line, when an x-fern or
x-stainless name appears in src/, tests/ or packages/forge/ outside
src/compat/ and tests/compat/. `just boundary` runs it.
- Each table checks its prefix (x-forge-, x-fern-, x-stainless-) before
  matching, and only x- keys are resolved, so a skipped key costs one
  compare instead of a match per table.
- A Hit holds only its field and slot; a key's name comes from the slot,
  in the per-kind name tables, when an error needs it. The reader's
  per-key result is back to 16 bytes, which fits in registers.

Output and the model dump of the aggregate are unchanged.
Builds REV in a throwaway worktree and runs the parse bench of both in
ABBA order, so drift in the machine's speed cancels, leaving out pairs in
which it changed speed. `just bench-compare [rev] [pairs]` runs it.
…tive

- x-forge-method-name, in line with the other -name keys. x-forge-method
  never shipped, so it gets no alias.
- x-forge-sensitive replaces x-sensitive, so every native key begins with
  x-forge-. x-sensitive stays readable as a deprecated native alias: a new
  Tier (native, deprecated, compat) decides which key's value is kept, and
  the alias works with compat off.
- tests/boundary.rs also catches vendor names spelled as identifiers
  (x_fern_...), and the five test names that still had them are renamed.

Output and the model dump of the aggregate are unchanged.
…ecision number

- Rule 4 now says per key what an empty or punctuation-only value does:
  names fall back to the wire name; an empty method name or group path is
  no method, and a punctuation-only one is an error.
- The npm release adds no compat module: the engine binary it ships is
  built with the default compat feature.
- "the D4 rule" is resolution rule 2; the D-numbers were never defined.
Today x-forge-group: [] makes no method. The intended meaning is a method
at the root, which needs root methods in the model, the WIT and the
generators, so it is documented as a known gap rather than fixed here.
… checked

- In an alias, the native method name, group, availability and ignore
  win over Fern's, in either order.
- A Fern group path or method name that is an error on its own is not
  checked when a native key wins.

Breaking the precedence rule fails all three.
- The boundary goal covers the engine; the docs packages read vendor keys
  themselves and are a non-goal.
- resolve.rs is the only place link calls compat, not the only cfg point.
- Deleting the layer lists every step, and what it leaves behind.
- With compat off, vendor keys become ordinary extensions for plugins.
- The name lookups are a second way vendor key names leave compat, as
  data, for errors.
- Test counts, the mutation result, and performance and memory are
  re-measured on the final code.
@Swimburger
Swimburger force-pushed the rfc-0001-metadata-resolution-layer branch from c6fd809 to e84e0e9 Compare October 26, 2026 21:04

@Swimburger Swimburger left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I think we should rename x-forge-group to x-forge-group-name for now, keep it more consistent with Fern until we rename it to x-forge-namespace.
  2. Are we unnecessarily slowing down CI at all?

Comment thread benches/compare.sh Outdated
Comment thread rfcs/0001-metadata-resolution-layer.md Outdated
Comment thread rfcs/0001-metadata-resolution-layer.md Outdated
Comment thread src/link/openapi.rs
Comment thread src/link/openapi.rs
- Rename x-forge-ignore to x-forge-skip, internal names included.
- Rename x-forge-group to x-forge-group-name, to match the other -name
  keys and stay apart from the legacy x-forge-group-info.
- Remove benches/compare.sh and its just recipe; the RFC describes the
  method inline.
- RFC: record why the key is model-name, and that CI compiles but does
  not test the build without compat, on purpose.
@Swimburger

Copy link
Copy Markdown
Contributor Author
  1. Done in 471acdf: x-forge-group-name, which also keeps it apart from the legacy x-forge-group-info. The RFC notes FORGE-3 expects to rename it to x-forge-namespace.
  2. No meaningful slowdown. The two new test binaries (boundary, compat) add a few seconds of compile and link, and their tests run in about 0.02 s. Testing the compat-off build is deliberately left out of CI to avoid a second full compile; clippy still compiles it. That is documented in the RFC (Enforcement).

@Swimburger
Swimburger marked this pull request as ready for review October 26, 2026 22:35
@Swimburger

Swimburger commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

@Swimburger Double check that downstream dependencies still receive raw extensions, cf CLI seems to depend on x-fern-... directly.
Edit: verified

@dimitropoulos dimitropoulos left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love it! one thing I'd love to keep up-to-date is the AGENTS.md file for this work. I'd love to keep it clear that the intention is that any compat flags are intended to be sectioned off into their own little area of the codebase

Comment thread rfcs/0001-metadata-resolution-layer.md
Comment thread src/compat/fern.rs Outdated
Comment thread src/compat/stainless.rs Outdated
Comment thread src/compat/fern.rs Outdated
Comment thread src/metadata/mod.rs Outdated
AGENTS.md gets a short section: only src/compat/ names vendor extensions,
it sits behind the compat feature, native keys win, and tests/boundary.rs
enforces it. src/compat/README.md holds the longer how-to, next to the code
it describes and deleted along with it.
Each level (operation or alias, parameter, schema, document) now has its own
module holding its key constants, its table of names and its reader, in the
native vocabulary and in each adapter. A key constant is no longer in scope
at a level it does not belong to. Tests check that every key is read at
exactly the levels that list it.

Also drop the key tables from the adapters' module docs: the code is the
source of truth for the keys.

@srikrsna srikrsna left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like the compatibility layer to be exposed as a separate command so that the spec we produce itself can be transformed. It will also reduce the complexity we have introduced to account for metadata resolution. That way the core will always parse just the forge extensions. And compat can truly be a separate layer altogether.

Comment thread rfcs/0001-metadata-resolution-layer.md Outdated
Comment thread rfcs/0001-metadata-resolution-layer.md Outdated
Comment thread rfcs/0001-metadata-resolution-layer.md Outdated
Comment thread rfcs/0001-metadata-resolution-layer.md Outdated
Comment thread rfcs/0001-metadata-resolution-layer.md
Drop the status header, the known gaps and follow-ups sections, the test plan
and the release shape, and shorten the rest. The vocabulary, the resolution
rules, the architecture and the alternatives considered stay.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants