RFC 0001: metadata resolution layer (FORGE-1)

AGENTS.md

@@ -8,6 +8,16 @@ API documentation site for Forge OpenAPI specs, built on Astro.
 - **`packages/fern-forge`**: `astro-fern` extension that validates and adapts Forge OpenAPI metadata.
 - **`packages/docs-site`**: API documentation website consuming `astro-fern` and `fern-forge`. The OpenAPI spec is fetched from the `openapi.forge.json` GitHub release asset (or a local file via `FORGE_OPENAPI_SPEC`).
 - **`packages/cloudflare-terraform-docs`**: Generator for the site's Terraform documentation data, derived from the released Cloudflare Terraform provider. Temporary: removed once Forge supports Terraform natively (see the site's README).
+- **`src/`**: The Rust engine: OpenAPI to linked model to plugin host. It reads only native `x-forge-*` extensions (`src/metadata/`); other tools' extensions are translated in `src/compat/`.
+
+## Vendor compatibility
+
+Only `src/compat/` knows vendor extensions (`x-fern-*`, `x-stainless-*`). It is sectioned off so it can be deleted as a whole. See [`src/compat/README.md`](src/compat/README.md).
+
+- Never name a vendor extension outside `src/compat/` and `tests/compat/`, comments and docs included. `tests/boundary.rs` enforces this (`just boundary`).
+- Compat is behind the default-on `compat` feature and is only a fallback: native `x-forge-*` keys always win.
+- Support for a new vendor key means mapping it onto an existing native primitive in an adapter. Add the native key first if there is none.
+- The docs packages (`astro-fern`, `fern-forge`, `cloudflare-api-site`) read vendor keys themselves and are not covered by the boundary.
 
 ## Conventions
 

Cargo.toml

@@ -23,7 +23,11 @@ name = "parse"
 harness = false
 
 [features]
-default = ["cli"]
+default = ["cli", "compat"]
+# The compatibility layer: it reads the extensions other tools' metadata is
+# written with in place of the native `x-forge-*` ones, when a document has
+# no native one. Without it, only the native extensions mean anything.
+compat = []
 # The `forge` command. Library users can turn it off with
 # `default-features = false`.
 cli = ["dep:clap", "dep:serde_json"]
@@ -49,6 +53,10 @@ required-features = ["cli"]
 name = "wasm"
 required-features = ["cli", "wasm"]
 
+[[test]]
+name = "compat"
+required-features = ["compat"]
+
 
 [lints.clippy]
 # https://rust-lang.github.io/rust-clippy/rust-1.88.0/index.html

justfile

@@ -27,6 +27,10 @@ lint:
     cargo clippy --locked --all-targets --all-features -- -D warnings
     cargo clippy --locked --all-targets --no-default-features -- -D warnings
 
+# Check that no other tool's extension is named outside the compatibility layer
+boundary:
+    cargo test --locked --test boundary
+
 # Release builds
 #
 # Every way of getting release binaries puts them in target/binaries/<target>/

packages/forge/plugin/forge.wit

@@ -82,7 +82,7 @@ interface forge {
         json: option<u32>,
     }
 
-    /// The stage of the API a method is in, from `x-fern-availability`.
+    /// The stage of the API a method is in, from `x-forge-availability`.
     enum availability {
         alpha, beta, preview, generally-available, deprecated,
     }

packages/forge/src/index.ts

@@ -144,7 +144,7 @@ export interface Method {
    * way around.
    */
   readonly deprecated: boolean;
-  /** The stage of the API the method is in, from `x-fern-availability`, or undefined if it doesn't say. */
+  /** The stage of the API the method is in, from `x-forge-availability`, or undefined if it doesn't say. */
   readonly availability: Availability | undefined;
   /**
    * Whether to leave the method out of listings of methods, such as a CLI's
@@ -197,7 +197,7 @@ export interface Method {
  */
 export type Location = { readonly in: 'path' } | { readonly in: 'query'; readonly exploded: boolean };
 
-/** The stage of the API a method is in, as `x-fern-availability` writes it. */
+/** The stage of the API a method is in, as `x-forge-availability` writes it. */
 export type Availability = 'alpha' | 'beta' | 'preview' | 'generally-available' | 'deprecated';
 
 /**
@@ -267,7 +267,7 @@ export interface Field {
    * though the same model's field appears in responses: `readOnly`.
    */
   readonly readOnly: boolean;
-  /** Whether the value is a secret, such as a password or a token, to keep out of logs: `x-sensitive`. */
+  /** Whether the value is a secret, such as a password or a token, to keep out of logs: `x-forge-sensitive`. */
   readonly sensitive: boolean;
   /** The `default` of the property's schema, or undefined if it has none. */
   readonly default: Json | undefined;

rfcs/0001-metadata-resolution-layer.md

@@ -0,0 +1,103 @@
+# RFC 0001: The metadata resolution layer
+
+## Summary
+
+The engine reads vendor metadata extensions (`x-fern-*`, `x-stainless-skip`) directly, so everything downstream of the linked model inherits the vendors' vocabulary and formats. This RFC gives the engine a native `x-forge-*` vocabulary and moves all vendor knowledge into one quarantined, feature-gated **compatibility layer** (`src/compat/`) that maps vendor keys onto native primitives. The layer is designed to be deleted as a whole.
+
+Output for existing specs is unchanged, and the release is non-breaking.
+
+## Motivation
+
+Vendor names are hardcoded in the serde key enums in `src/link/openapi.rs`, in value formats (Fern's availability words, Stainless's skip list), in validation messages, in the list of keys kept from plugins, and in the public docs, the WIT and the JS runtime. Anything downstream, including the cf codebase, depends on a vendor's extension format, and adding another vendor means touching all of it again.
+
+The released aggregate uses the keys forge reads about 11,400 times (`x-fern-sdk-*` 3,638 each, `x-fern-availability` 3,444, `x-fern-ignore` 449, `x-stainless-skip` 114, and about 150 name keys). It also carries keys forge does not read, such as `x-fern-enum` (3,652) and `x-stainless-terraform-configurability` (407).
+
+## Scope
+
+Goals:
+
+- Nothing in the engine (`src/`, `tests/`, `packages/forge`) outside `src/compat/` and `tests/compat/` names a vendor extension, enforced by a test.
+- Output for existing specs is byte-identical; no meaningful parse-time regression.
+- Native keys are applied first; the compatibility layer is only a fallback, and can be removed without changing what any `x-forge-*` key means.
+
+Non-goals: Fern's structural behavior (allOf/anyOf, inline request types), operation ID prefixing, the cf codebase's own Fern-to-Forge rename, dotted `operationId` derivation, and the docs packages in this repository, which read vendor keys themselves. Vendor keys with no Forge primitive (`x-fern-enum`, `x-fern-audiences`, `x-stability` and others) stay unread.
+
+## The native vocabulary
+
+| Native key | Value | Meaning | Compat source |
+|---|---|---|---|
+| `x-forge-method-name` | string | the method's name | `x-fern-sdk-method-name` |
+| `x-forge-group-name` | dotted string, or array of strings, top level first | the group path; an empty path makes no method, as today | `x-fern-sdk-group-name`, read the same way |
+| `x-forge-skip` | bool | operation or alias: no method; property schema: no field | `x-fern-ignore` (operation), `x-stainless-skip` (property; a list of targets counts as `false`) |
+| `x-forge-availability` | `alpha`, `beta`, `preview`, `generally-available`, `deprecated` | lifecycle stage | `x-fern-availability` |
+| `x-forge-model-name` | string | the model's name | `x-fern-type-name` |
+| `x-forge-property-name` | string | the property's field name | `x-fern-property-name` |
+| `x-forge-parameter-name` | string | the parameter's field name | `x-fern-parameter-name` |
+| `x-forge-sensitive` | bool | the value is a secret, to mask | none; replaces `x-sensitive` |
+| `x-sensitive` | bool | deprecated name of `x-forge-sensitive`, read only where that is not written | none |
+
+`x-forge-hidden`, `x-forge-require-confirmation`, `x-forge-aliases` and `x-forge-commands` are unchanged.
+
+Design notes:
+
+- **Explicit keys, not one positional name key.** Tools that inline `$ref` targets would make a positional `x-forge-name` silently rename fields. A misplaced explicit key can be ignored or flagged instead.
+- **`model-name`** sets `Model::name`, the word the Rust API, the WIT and the JS runtime use. `type-name` would promise more than it does, `schema-name` clashes with `Model::schema_name`, and `class-name` and `title` are a language's word and OpenAPI's display string.
+- **`skip`**, not `ignore`: it says what happens, and stands apart from `x-forge-hidden` (generated, but not listed). One word covers operations and properties.
+- **Groups as a dotted string or an array.** Cloudflare's specs write dotted strings, so moving to the native key is a rename with the value unchanged. Fern reads a string as one group and never splits on dots, so for a string with a dot the adapter and Fern disagree. That is deliberate. FORGE-3 expects to rename the key to `x-forge-namespace`.
+- **Every native key begins with `x-forge-`.** `x-sensitive` predates that rule: it stays readable (155 production uses) as a deprecated alias, which is part of the native vocabulary and keeps working with compat off.
+- **No misplacement validation.** A native key at the wrong level is silently ignored, as Fern's keys are today. Validating later only tightens.
+
+## Resolution rules
+
+1. Keys rank in three tiers: native, then deprecated native alias, then compat. The strongest tier present wins, per primitive and whatever order the keys are written in. Only the winning value is validated.
+2. The same key written twice is an error. A value of the wrong JSON type is an error whichever key it is on.
+3. Names with no letters or digits behave as they always have: a property, parameter or model name is ignored, a method name or group path that is empty makes no method, and a non-empty one that has none is an error.
+4. Errors name the key the author actually wrote.
+5. Writing native and compat keys with the same value is the supported way to stay portable across engine versions during the transition.
+
+A consequence of rule 1: a stale compat key silently diverges from a native key written beside it.
+
+## Architecture
+
+```
+src/
+├─ metadata/   the native table, per kind of object (definition, parameter,
+│              schema, document): keys, values, formats. Pure: `&str` in,
+│              small values out. Imports neither compat nor link.
+├─ compat/     the adapters (fern, stainless) and their aggregator. The only
+│              code holding vendor keys. Behind the default-on `compat` feature.
+└─ link/
+   ├─ resolve.rs  native first, then compat; the only place link calls compat
+   └─ openapi.rs  OpenAPI's own keys; every `x-` key asks resolve.rs
+```
+
+- Dependencies: `link` uses `metadata` and, behind the feature, `compat`; `compat` uses `metadata`.
+- The layer maps keys to primitives and normalizes simple values. Structural behavior, such as aliases becoming methods, stays in the builder.
+- Without the `compat` feature, vendor keys are ordinary extensions and are passed to plugins.
+- Deleting the layer means removing `src/compat/`, `tests/compat/`, `mod compat` in `src/lib.rs`, the compat arms in `link/resolve.rs`, and the feature in `Cargo.toml`. No native key changes. A little native-side code that only the layer used is left over and can be removed afterwards.
+
+## Enforcement
+
+- Every vendor key literal is private to its adapter, and the compiler enforces that.
+- `tests/boundary.rs` searches `src/`, `tests/` and `packages/forge/` (docs and WIT included) for `x-fern` and `x-stainless` outside the layer.
+- CI compiles the crate without `compat` through clippy, but runs tests only with all features, to avoid a second full compile for a configuration nothing ships.
+
+## Verification
+
+Checked against the unmodified engine on the released aggregate (27.6 MB):
+
+- The original 379 Fern-keyed parse tests passed before any fixture was migrated.
+- `forge --target ts` writes the same 3,381 files byte for byte, and the full model dump (197,702 lines) is identical, natively and through the WASM plugin boundary.
+- `cargo test` passes with compat on, and with it off when run locally; clippy is clean in the library-only, compat-off and all-features configurations.
+- Parse time (divan `parse` bench, paired base-new-new-base runs) is 0.8% slower (+0.34 ms of 42.0 ms, t = 1.9). Peak memory is +0.07%, with the same allocation count.
+
+## Alternatives considered
+
+- **Resolve after parse** (keep every `x-` key as raw JSON, normalize in a second pass): it must retain every extension on the 27 MB aggregate unless the reader consults a key table anyway, which collapses into this design. Rejected on memory and diff size.
+- **A `Vendor` trait** with an implementation per tool: no primitive overlaps across vendors today, so the table just grows rows. Over-engineering for now.
+- **Separate words for operations and properties** (`ignore` and `skip`): two words for one concept.
+- **Arrays only for `x-forge-group-name`**: every migration would reshape values as well as rename keys.
+- **Reading a string group as one group, as Fern does**: the native and Fern keys would read the same value differently.
+- **Erroring when native and compat keys disagree**: stricter, but costs comparison machinery for no gain on correct specs. Silent native-wins was chosen.
+- **A separate `metadata` crate now**: it would pull `Availability` and the other primitive types out of the engine. Deferred until a standalone consumer exists; the pure contract keeps that move mechanical.
+- **Renaming `Group` and `Availability` now**: both break the plugin contract, so both wait for FORGE-3's single breaking release.

src/compat/README.md

@@ -0,0 +1,47 @@
+# The compatibility layer
+
+The engine understands one vocabulary: the native `x-forge-*` extensions in
+[`src/metadata/`](../metadata/mod.rs). This folder is the only place that
+knows any other tool's extensions (Fern's `x-fern-*`, Stainless's
+`x-stainless-*`). Each adapter maps its tool's keys onto native primitives.
+
+The layer is meant to be deleted as a whole once nobody writes those keys.
+Everything below follows from that.
+
+## Rules
+
+- **Only this folder names a vendor extension.** That includes the readers,
+  the linker, the plugin host, the WIT, the `packages/forge` npm package, and
+  their comments and docs. `tests/boundary.rs` fails, with file and line, if
+  one appears elsewhere (`just boundary`; it also runs with `cargo test`).
+  Vendor tests live in `tests/compat/`.
+- **Native wins.** A vendor key is read only for a primitive the document
+  has no native key for. When both are written, the native value is used and
+  the vendor value is not even validated, whatever order they appear in.
+- **Behind the `compat` feature** (on by default). Native keys must mean the
+  same with it off. CI compiles that configuration (clippy with
+  `--no-default-features`) but does not run its tests.
+- **Mapping, not semantics.** An adapter says which native primitive a key
+  stands for and how its value is written, as data. Anything structural, such
+  as how aliases become methods, belongs to the builder, not here.
+- **No vendor knowledge leaks out.** Keys are private to their adapter.
+  `mod.rs` is the only way in: one resolver per kind of object, and one name
+  lookup per kind so errors can name the key the author wrote.
+
+## Adding a vendor key
+
+1. Find the native primitive it stands for in `src/metadata/`. If there is
+   none, add a native `x-forge-*` key first: a vendor key never introduces a
+   primitive.
+2. Map it in the adapter for its tool, or add an adapter next to `fern.rs`
+   and `stainless.rs` and register it in `mod.rs`.
+3. Add a test in `tests/compat/` showing that the vendor-keyed document
+   parses to the same `Forge` as the native-keyed one.
+
+## Removing the layer
+
+Delete this folder and `tests/compat/`, `mod compat` in `src/lib.rs`, the
+`compat` arms in `src/link/resolve.rs`, and the `compat` feature and its
+`[[test]]` entry in `Cargo.toml`. No native extension changes. A few
+native-side items exist only for the layer and are left unused; removing them
+is a mechanical follow-up.