feat!: forge reads native extension keys only, with forge normalize on serde (FORGE-1)

AGENTS.md

@@ -8,6 +8,16 @@ API documentation site for Forge OpenAPI specs, built on Astro.
 - **`packages/fern-forge`**: `astro-fern` extension that validates and adapts Forge OpenAPI metadata.
 - **`packages/docs-site`**: API documentation website consuming `astro-fern` and `fern-forge`. The OpenAPI spec is fetched from the `openapi.forge.json` GitHub release asset (or a local file via `FORGE_OPENAPI_SPEC`).
 - **`packages/cloudflare-terraform-docs`**: Generator for the site's Terraform documentation data, derived from the released Cloudflare Terraform provider. Temporary: removed once Forge supports Terraform natively (see the site's README).
+- **`src/`**: The Rust engine: OpenAPI to linked model to plugin host. It reads only native `x-forge-*` extensions (`src/metadata/`); other tools' extensions are rewritten into them by `forge normalize` (`src/normalize/`).
+
+## Vendor extensions
+
+Compatibility with other tools' extensions (`x-fern-*`, `x-stainless-*`, and `x-sensitive`) is sectioned off into its own area, `src/normalize/vendor/`, and nowhere else in the codebase knows them. `forge` does not read them: they are ordinary extensions to it, and `forge normalize` is what rewrites them into native ones. Any new compatibility key belongs in that area. See [`src/normalize/vendor/README.md`](src/normalize/vendor/README.md).
+
+- Never name a vendor extension outside `src/normalize/vendor/` and `tests/normalize/`, comments and docs included. `tests/boundary.rs` enforces this (`just boundary`).
+- Run `forge normalize --write --leave-old-extensions` on a vendor-keyed document before `forge` reads it. The old keys stay for tools that still read them. Until the release that makes this step required, the `forge` command also does it in memory when it reads a document (`before_reading` in `src/bin/forge/normalize.rs`); that call is temporary, and the engine in `src/` never does it.
+- Support for a new vendor key means mapping it onto an existing native primitive in an adapter. Add the native key first if there is none.
+- The docs packages (`astro-fern`, `fern-forge`, `cloudflare-api-site`) read vendor keys themselves and are not covered by the boundary.
 
 ## Conventions
 

Cargo.toml

@@ -8,15 +8,15 @@ edition = "2024"
 
 [dependencies]
 clap = { version = "4.6.7", features = ["derive"], optional = true }
+memchr = "2.8.3"
 rayon = "1.12.0"
 serde = { version = "1.0.229", features = ["derive"] }
-serde_json = { version = "1.0.151", optional = true }
+serde_json = { version = "1.0.151", features = ["preserve_order"] }
 wasmtime = { version = "49.0.2", default-features = false, features = ["runtime", "cranelift", "component-model", "parallel-compilation", "std"], optional = true }
 wasmtime-wasi = { version = "49.0.2", default-features = false, features = ["p2"], optional = true }
 
 [dev-dependencies]
 divan = "0.1.21"
-serde_json = "1.0.151"
 
 [[bench]]
 name = "parse"
@@ -26,7 +26,7 @@ harness = false
 default = ["cli"]
 # The `forge` command. Library users can turn it off with
 # `default-features = false`.
-cli = ["dep:clap", "dep:serde_json"]
+cli = ["dep:clap"]
 # `forge::wasm`, which runs plugins compiled to WebAssembly components.
 wasm = ["dep:wasmtime", "dep:wasmtime-wasi"]
 

justfile

@@ -27,6 +27,10 @@ lint:
     cargo clippy --locked --all-targets --all-features -- -D warnings
     cargo clippy --locked --all-targets --no-default-features -- -D warnings
 
+# Check that no other tool's extension is named outside the vendor tables of `forge normalize`
+boundary:
+    cargo test --locked --test boundary
+
 # Release builds
 #
 # Every way of getting release binaries puts them in target/binaries/<target>/

packages/forge/README.md

@@ -16,6 +16,80 @@ npx @cloudflare/forge openapi.json --target ts --out sdk
 Pass `-` instead of a path to read the document from standard input. Run
 `forge --help` for every option.
 
+## Normalizing a document
+
+Forge's own metadata is the `x-forge-*` extensions, and they are the only ones
+it reads. Documents written for other tools carry those tools' extensions
+instead, and `forge normalize` rewrites them into the native ones. Run it
+before forge reads the document:
+
+```sh
+# Print the document with its keys rewritten. Nothing is changed.
+npx forge normalize openapi.json
+
+# Rewrite the files in place.
+npx forge normalize --write specs/*.json
+
+# Keep the old keys beside the native ones, for what still reads them.
+npx forge normalize --write --leave-old-extensions openapi.json
+
+# Write nothing, and fail if anything would be rewritten: for a check in CI.
+npx forge normalize --check specs/*.json
+
+# Read from standard input, write to standard output, and generate from that.
+cat openapi.json | npx forge normalize - | npx forge - --target ts --out sdk
+```
+
+Without `--write` it changes nothing and prints one document to standard
+output, so several files need `--write` or `--check`. A file with nothing to
+rewrite is left as it is, a link is followed and stays a link, and a read-only
+file is left as it is, with an error. `-v` says how many times each key was
+rewritten. Documents have to be JSON: YAML is not supported yet.
+
+What is rewritten:
+
+- A group path written as an array becomes the names joined by dots
+  (`["a", "b"]` becomes `"a.b"`), unless a name has a dot in it.
+- A skip for only some targets, written as a list, has no native form, since
+  `x-forge-skip` is for every target: it is left as it is. A skip that is plainly
+  true or false is rewritten.
+- Native keys always win: when a document says the same thing twice, the native
+  key is kept, and the other is reported as a warning if it says something else.
+
+For now, `forge` also does this in memory when it reads a document, so one that
+has not been through `forge normalize` still builds. It keeps the old keys, as
+`--leave-old-extensions` does, so plugins read them as ordinary extensions. That
+stops in the release that makes `forge normalize` a step to run first.
+
+<details>
+<summary>Known limitations</summary>
+
+`forge normalize` reads a document as JSON and writes it again, so a file that
+has a key to rewrite is not a minimal edit. The order of its keys is kept.
+
+- **A key written twice** keeps only its last copy, with no error or warning.
+  `forge` itself rejects such a key, and no longer does in a document that has a
+  key to rewrite.
+- **Numbers** that do not fit in 64 bits are rounded (`100000000000000000001`
+  becomes `1e+20`), a decimal of more than about 17 digits is shortened, and
+  `1E5`, `0.10` and `-0` are written `100000.0`, `0.1` and `-0.0`.
+- **Formatting** is the writer's: the indentation of the first indented line is
+  used for the whole file, arrays and objects on one line are written on
+  several, blank lines, spaces at the end of a line and anything before the
+  first `{` are dropped, `\r\n` becomes `\n`, and escapes such as `\u00e9` and
+  `\/` are written as the characters.
+- **Error positions**: an error `forge` finds later in a document that has a key
+  to rewrite is at a line and column of the rewritten document, which can be past
+  the end of the file. `forge` says so, and `forge normalize
+--leave-old-extensions` prints that document.
+- **Errors**: JSON with comments, a byte order mark, and a document nested
+  deeper than 128 levels are not read. A key written with an escape is rewritten
+  by `forge normalize`, and not by the step `forge` does in memory.
+- **Memory**: a document is held in memory at about 6.5 times its size, one file
+  at a time.
+
+</details>
+
 ## Plugins
 
 A plugin generates an SDK, or anything else, from the document: forge parses

packages/forge/plugin/forge.wit

@@ -82,7 +82,7 @@ interface forge {
         json: option<u32>,
     }
 
-    /// The stage of the API a method is in, from `x-fern-availability`.
+    /// The stage of the API a method is in, from `x-forge-availability`.
     enum availability {
         alpha, beta, preview, generally-available, deprecated,
     }

packages/forge/src/index.ts

@@ -144,7 +144,7 @@ export interface Method {
    * way around.
    */
   readonly deprecated: boolean;
-  /** The stage of the API the method is in, from `x-fern-availability`, or undefined if it doesn't say. */
+  /** The stage of the API the method is in, from `x-forge-availability`, or undefined if it doesn't say. */
   readonly availability: Availability | undefined;
   /**
    * Whether to leave the method out of listings of methods, such as a CLI's
@@ -197,7 +197,7 @@ export interface Method {
  */
 export type Location = { readonly in: 'path' } | { readonly in: 'query'; readonly exploded: boolean };
 
-/** The stage of the API a method is in, as `x-fern-availability` writes it. */
+/** The stage of the API a method is in, as `x-forge-availability` writes it. */
 export type Availability = 'alpha' | 'beta' | 'preview' | 'generally-available' | 'deprecated';
 
 /**
@@ -267,7 +267,7 @@ export interface Field {
    * though the same model's field appears in responses: `readOnly`.
    */
   readonly readOnly: boolean;
-  /** Whether the value is a secret, such as a password or a token, to keep out of logs: `x-sensitive`. */
+  /** Whether the value is a secret, such as a password or a token, to keep out of logs: `x-forge-sensitive`. */
   readonly sensitive: boolean;
   /** The `default` of the property's schema, or undefined if it has none. */
   readonly default: Json | undefined;

rfcs/0001-metadata-resolution-layer.md

@@ -0,0 +1,75 @@
+# RFC 0001: Native metadata, and `forge normalize`
+
+## Summary
+
+`forge` reads a native `x-forge-*` vocabulary and nothing else. The extensions other tools write (`x-fern-*`, `x-stainless-skip`, `x-sensitive`) are ordinary extensions to it. A separate command, `forge normalize`, rewrites them into the native ones, so the spec that is produced is itself what gets transformed, and the compatibility knowledge is a layer of its own.
+
+## Motivation
+
+Vendor names were hardcoded in the readers, in value formats, in error messages, in the keys kept for plugins and in the public docs, so everything downstream inherited a vendor's vocabulary, and adding a vendor meant touching all of it. Reading them at runtime, even as a fallback, also means a ranking between keys and a vendor key that silently diverges from the native key beside it. Rewriting them in an explicit step makes the change visible as a diff, and reports a conflict.
+
+## Scope
+
+- `forge` reads native keys only: no tiers, no fallback, no feature to turn on.
+- Nothing outside `src/normalize/vendor/` and `tests/normalize/` names a vendor extension, enforced by a test.
+- A document, after `forge normalize`, builds what it built before.
+
+Not in scope: Fern's structural behavior (allOf/anyOf, inline request types), operation ID prefixing, and the docs packages, which read vendor keys themselves. Vendor keys with no Forge primitive are neither read nor rewritten.
+
+## The native vocabulary
+
+| Native key               | Value                                                           | Meaning                                           | Rewritten from                                             |
+| ------------------------ | --------------------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------- |
+| `x-forge-method-name`    | string                                                          | the method's name                                 | `x-fern-sdk-method-name`                                   |
+| `x-forge-group-name`     | dotted string, or array of strings                              | the group path; an empty path makes no method     | `x-fern-sdk-group-name`                                    |
+| `x-forge-skip`           | bool                                                            | operation or alias: no method; property: no field | `x-fern-ignore` (operation), `x-stainless-skip` (property) |
+| `x-forge-availability`   | `alpha`, `beta`, `preview`, `generally-available`, `deprecated` | lifecycle stage                                   | `x-fern-availability`                                      |
+| `x-forge-model-name`     | string                                                          | the model's name                                  | `x-fern-type-name`                                         |
+| `x-forge-property-name`  | string                                                          | the property's field name                         | `x-fern-property-name`                                     |
+| `x-forge-parameter-name` | string                                                          | the parameter's field name                        | `x-fern-parameter-name`                                    |
+| `x-forge-sensitive`      | bool                                                            | the value is a secret, to mask                    | `x-sensitive`, which other systems write too               |
+
+`x-forge-hidden`, `x-forge-require-confirmation`, `x-forge-aliases` and `x-forge-commands` are unchanged.
+
+- Explicit keys, not one positional name key: tools that inline `$ref` targets would make a positional key silently rename fields.
+- `skip`, not `ignore`: it says what happens, and stands apart from `x-forge-hidden`. One word covers operations and properties.
+- Groups are a dotted string or an array, so a move from Cloudflare's dotted strings is a rename. A string is names separated by dots; an array is the names, which are not split. FORGE-3 expects to rename the key to `x-forge-namespace`.
+- A native key at the wrong level is ignored. Validating later only tightens.
+
+## What `forge normalize` does
+
+- It changes nothing unless it is told to: one document is printed, rewritten, to standard output, and `--write` rewrites each file in place, only if something in it changes. `-` reads standard input and prints. `--check` writes nothing and fails if anything would change.
+- A key is rewritten only at the kind of object `forge` reads its native key at: an operation or an alias, a parameter, or a schema.
+- The document is read and written again, with its key order and the indentation of its first indented line kept. That is simple, and it is not a minimal edit: what it changes beyond the keys is listed under "Known limitations" in the README of the package.
+- A native key is never replaced. When a document says the same thing twice, the native key is kept, and the other is dropped, or reported if it says something else.
+- A group path that is an array becomes the names joined by dots, unless a name has a dot in it.
+- A skip on a property is rewritten to `x-forge-skip` with the same value: `x-stainless-skip: true` becomes `x-forge-skip: true`. `x-forge-skip` is for every target, so a skip for only some targets, written as a list (`x-stainless-skip: ["terraform"]`), has no native form and is left as it is written, in both modes.
+- `--leave-old-extensions` keeps each key that is rewritten, beside the native one, for what still reads it.
+
+## Architecture
+
+```
+src/
+├─ metadata/        the native table, per kind of object: definition,
+│                   parameter, schema, document. Pure: `&str` in, small
+│                   values out.
+├─ link/            reads native keys through `metadata`.
+└─ normalize/
+   ├─ vendor/       the tables of other tools' keys. The only code holding
+   │                them, sectioned off, and per level.
+   └─ walk.rs       where a key is to be rewritten, and to what.
+```
+
+## Migration
+
+- Run `forge normalize --write` before `forge` reads a document written with vendor keys, with `--leave-old-extensions` where something else still reads them. A pipeline is `cat spec.json | forge normalize - | forge - --target ts --out sdk`.
+- Until the release that makes `forge normalize` a step to run first, the `forge` command does the same rewrite in memory when it reads a document, so one that has not been through it builds as before. It leaves the old keys, as `--leave-old-extensions` does, so plugins read them as the ordinary extensions they become in that release. It costs a pass over the document, and stops in that release; the core never reads anything but native keys.
+- After that release, a document that is not normalized is not an error: `forge` finds no method names in it, and so no methods.
+- Keys that are left are ordinary extensions, so plugins see them.
+
+## Alternatives considered
+
+- **A runtime fallback behind a feature**, native keys first and vendor keys for what has none: it costs tiers in the reader and a vendor key that diverges from the native one beside it. Rejected for an explicit rewrite.
+- **Resolving after parsing, in the engine**: it must retain every extension of a large document.
+- **Normalizing in memory before every `generate`, for good**: a pass over the document on every run, to save one in CI. Kept for now as a bridge, and not as the design.
+- **A `Vendor` trait** per tool: no primitive overlaps across vendors, so the table only grows rows.