feat!: forge reads native extension keys only (FORGE-1) by Swimburger · Pull Request #73 · cloudflare/forge

Swimburger · 2026-10-28T00:52:01Z

forge reads the native x-forge-* keys and nothing else, and a new command, forge normalize, rewrites the extensions other tools write (x-fern-*, x-stainless-skip, x-sensitive) into them. FORGE-1.

This is one PR against main: it carries the native vocabulary of RFC 0001 and forge normalize, so it does not depend on any other PR.

What

A native vocabulary, kept by src/metadata/ per kind of object, described in rfcs/0001-metadata-resolution-layer.md: x-forge-method-name, x-forge-group-name (a dotted string or an array), x-forge-skip, x-forge-availability, x-forge-model-name, x-forge-property-name, x-forge-parameter-name and x-forge-sensitive.

forge normalize rewrites the keys of other tools into those, in place:

forge normalize spec.json                         # prints the rewritten document, changes nothing
forge normalize --write specs/*.json              # rewrites the files in place (the shell expands the pattern)
forge normalize --check specs/*.json              # fails if anything would change
forge normalize --write --leave-old-extensions spec.json  # keep the old keys beside the native ones
cat spec.json | forge normalize - | forge - --target ts --out sdk

It changes nothing unless it is given --write: without it one document is printed, rewritten, to standard output. It edits the bytes of the document and does not write it again, so whitespace, key order and the text of every number stay as they were, and a diff is only the keys. A native key is never replaced, and a key is rewritten only at the kind of object forge reads its native key at.

The tables of other tools' keys are src/normalize/vendor/, per level, and are the only code that names them. A boundary test (tests/boundary.rs) enforces it, and AGENTS.md says that compatibility is sectioned off there.

Commits:

c012837: all of the above. The engine has no fallback, no tiers and no compat feature.
fe43db8: until the release that makes forge normalize a step to run first, the forge command runs the same rewrite in memory before it parses. Temporary, and only the binary does it. Reverting this commit removes it.
After review: d3af272 (docs), 1c349eb (forge::normalize is not behind a feature), b96f405 (the reader no longer tracks which key gave a value) and 5cbfa30 (forge normalize takes files, and the shell expands patterns) and 00f943d (forge normalize changes nothing unless it is given --write).

forge normalize reads JSON only for now. YAML is refused with a clear error, and is planned as a follow-up.

Breaking

A document written with other tools' keys builds only what its native keys say, unless it has been through forge normalize. The in-memory rewrite keeps the forge command working for now; Rust code that parses a Forge itself does not get it.

Performance

forge normalize on the 28 MB released aggregate, the whole process, best of 20:


first run, rewrites the file in place	26.7 ms
`--leave-old-extensions`, first run	27.9 ms
already normalized, nothing to write	20.2 ms
`--check`	23.2 ms
standard input to standard output	23.1 ms

The in-memory rewrite plus the parse, same document, best of 30:


rewrite, raw document (11,714 keys to rewrite)	17.6 ms
rewrite, normalized document (nothing to rewrite)	14.4 ms
parse into a `Forge`	41.6 ms

The whole forge --target ts run goes from about 252 ms to 262 ms for a normalized document and to 270 ms for a raw one, which is 7%. The rewrite walks the document once with no tree, and checks it to be JSON on a second thread.

Behavior to look at

x-sensitive is not read by the engine. forge normalize rewrites it to x-forge-sensitive. It is not called deprecated, since other systems write it, and --leave-old-extensions keeps it for them. @srikrsna: if the engine should keep reading it, that brings back some handling of two keys for one thing.
Keys that are left are ordinary extensions, so plugins see them. The default rewrite removes the keys it rewrites.
normalize is lenient, and the forge command is not. A key written twice, or a value that cannot be read, is left and warned about by forge normalize. The forge command fails on a key written twice with the error it always gave, duplicate field `x-fern-ignore` at line 1 column 174, and leaves a value that cannot be read for the engine to say.
An error forge finds later in the document says the column of the rewritten document, which differs from the one read where a key was renamed earlier on the same line. The line is the same. It goes with the in-memory rewrite.

Verification

On the released aggregate, against the SDK the engine wrote before the change:

--target ts writes all 3,243 files, byte for byte, from the raw document (through the in-memory rewrite), from the document after forge normalize in either mode, and through cat spec.json | forge normalize - | forge - --target ts. Without the rewrite, the raw document writes 7 files, so the check can tell.
Eight documents with a key written twice, in an operation, an alias, a parameter and a schema, and three with a value that cannot be read, give the same error as before, word for word, line and column included.
FORGE_SPEC=openapi.forge.json cargo test --release --test normalize aggregate -- --ignored builds the same model in both modes, with no conflicts.
The first two commits each build and pass the formatter, clippy (-D warnings) with all features and with none, and their tests (613 and 619). The final tree passes the same and also compiles for x86_64-pc-windows-msvc.

Follow-up

Remove the in-memory rewrite in the release that makes forge normalize required: revert fe43db8. Add YAML input to forge normalize.

…writes the rest `forge` reads a native `x-forge-*` vocabulary and nothing else. The extensions other tools write (`x-fern-*`, `x-stainless-skip`, `x-sensitive`) are ordinary extensions to it, and a new command, `forge normalize`, rewrites them into the native ones, so the spec that is produced is itself what gets transformed. FORGE-1. The native vocabulary: `x-forge-method-name`, `x-forge-group-name` (a dotted string or an array), `x-forge-skip`, `x-forge-availability`, `x-forge-model-name`, `x-forge-property-name`, `x-forge-parameter-name` and `x-forge-sensitive`, kept by `src/metadata/` per kind of object, alongside `x-forge-hidden`, `x-forge-require-confirmation`, `x-forge-aliases` and `x-forge-commands`. The reader asks it for a key and finds one written twice; there is no fallback and no ranking between keys. `forge normalize [--leave-old-extensions] [--check] [-v] <INPUT>...` takes files, folders (every .json file below them, except hidden ones and those in a node_modules), glob patterns, or - for standard input to standard output, and rewrites them in place. - It edits the bytes of the document and does not write it again, so whitespace, the order of keys and the text of every number stay as they were, and a diff is only the keys. It walks the document once with no tree, and checks it to be JSON on a second thread. - A key is rewritten only at the kind of object `forge` reads its native key at, so it never gets a meaning it did not have. The tables of other tools' keys are `src/normalize/vendor/`, per level, the only code that names them; a boundary test enforces it. - A native key is never replaced: when a document says the same thing twice, the native key is kept, and the other is dropped, or reported if it says something else. - A group path that is an array becomes the names joined by dots, unless a name has a dot in it. What no native key says, such as a skip that is a list of targets, is left as it is written. - What it cannot rewrite, a key written twice or a value that cannot be read, is left and warned about, with where a parser would stop. - Files are replaced by writing a new one beside them and renaming it, or in place where that is not possible, as for a hard link. A file with more than one name is rewritten once, and a link stays a link. - `--check` writes nothing and fails if anything would change. RFC 0001 describes the design, and AGENTS.md says that compatibility with other tools' extensions is sectioned off in `src/normalize/vendor/`. Breaking: a document written with other tools' keys builds only what its native keys say until it has been through `forge normalize`.

…or now Until the release that makes `forge normalize` a step to run first, `forge` runs the same rewrite on the bytes it reads, before it parses them, so a document that has not been through `forge normalize` builds what it did: the released aggregate writes the same 3,243 files of `--target ts`, byte for byte, without the step. FORGE-1. It is temporary, and only the binary does it: the engine in `src/` reads native keys and nothing else. - It is the default rewrite, so the keys it rewrites are not kept, as `forge` did not hand them on to plugins before. - It says nothing of what it leaves, as `forge` said nothing of the document before. A key written twice is an error in the words `forge` used when it read those keys itself, down to the line and column: "duplicate field `x-fern-sdk-method-name` at line 7 column 30". A value that cannot be read is left in place, and `forge` says it as it reads it. - An error `forge` finds later in the document gives the column of the document that was rewritten, which differs from the one that was read where a key was renamed earlier on the same line. The line is the same. Cost, on the 28 MB released aggregate, best of 30: the rewrite is 17.6 ms when it has 11,714 keys to rewrite and 14.4 ms when it has none, next to 41.6 ms to parse into a `Forge`. The whole `--target ts` run goes from about 252 ms to 262 ms for a normalized document and to 270 ms for a raw one, which is 7%.

srikrsna

Left some comments, I think we can let go of trying to retain whitespace changes, retaining the order is good enough. There is a mini json parser in this code which I would like to avoid. We should also lean on serde to help us support both json and yaml.

srikrsna · 2026-10-28T03:20:09Z

+## What `forge normalize` does
+
+- A key is rewritten only at the kind of object `forge` reads its native key at: an operation or an alias, a parameter, or a schema.
+- The document is edited as bytes, not written again: whitespace, key order and the text of every number stay as they were, so a diff is only the keys.


Do we even retain whitespace?

Yes, deliberately. This is for teams migrating their own specs, in whatever layout and with whatever comments they have, and they'll commit the result. Editing in place keeps their diff to just the renamed keys. Rewriting from a tree reformats any file that isn't already in serde's layout. Key order is kept either way.

YAML input isn't supported yet, but can be added as a fast follow.

srikrsna · 2026-10-28T03:22:01Z

+- The document is edited as bytes, not written again: whitespace, key order and the text of every number stay as they were, so a diff is only the keys.
+- A native key is never replaced. When a document says the same thing twice, the native key is kept, and the other is dropped, or reported if it says something else.
+- A group path that is an array becomes the names joined by dots, unless a name has a dot in it.
+- What no native key says, such as a skip that is a list of targets, is left as it is written.


I don't understand this line

Fair, poorly written. This says that x-stainless-skip: true is converted to x-forge-skip: true, but x-stainless-skip: ['terraform'] doesn't have a native forge equivalent, and is therefore untouched.

srikrsna · 2026-10-28T03:23:20Z

+    /// have every `.json` file below them rewritten, except those that are
+    /// hidden or in a node_modules, or as glob patterns such as
+    /// 'specs/**/*.json', which are taken as they are written. Or - to read one from standard input and write
+    /// it to standard output.


We should not account for node_modules here. They can pass anything they want, glob expansion is a shell concern.

Not all shells provide good support for that, but we don't have to worry about wider audience UX at this point. We can keep it simple for now.

srikrsna · 2026-10-28T03:24:46Z

+- Groups are a dotted string or an array, so a move from Cloudflare's dotted strings is a rename. A string is names separated by dots; an array is the names, which are not split. FORGE-3 expects to rename the key to `x-forge-namespace`.
+- A native key at the wrong level is ignored. Validating later only tightens.
+
+## What `forge normalize` does


Does forge normalize replace the input file?

It did when you made this comment, but I think it should not, so I made it require an explicit flag to do in place writes.

Without --write it prints the rewritten document to stdout and changes nothing; --write rewrites each file in place, and only if something in it changes. --check writes nothing and fails if anything would change, and - reads stdin. I've added this to the RFC.

srikrsna · 2026-10-28T03:25:47Z

+    #[cfg(feature = "normalize")]
+    use clap::Parser;
+    #[cfg(feature = "normalize")]


I don't think we need to feature gate this one

srikrsna · 2026-10-28T03:28:40Z

+/// A value an extension gives, and the key it came from, so that an error can
+/// name the key the document wrote.
+#[derive(Default)]
+pub(super) struct Val<T> {
+    pub(super) value: T,
+    /// The slot of the key that gave the value, which names it. It means
+    /// nothing if no key gave one.
+    pub(super) slot: u8,
+}
+
+impl<T> Val<T> {
+    /// Keeps `value`, which `hit` gave. No other key says the same thing, and
+    /// one written twice is an error before it gets here.
+    fn set<F>(&mut self, hit: &Hit<F>, value: T) {
+        self.value = value;
+        self.slot = hit.slot;
+    }
+}
+


From the RFC, we are only reading forge extensions so I don't think we need this anymore.

srikrsna · 2026-10-28T03:31:23Z

+}
+
+/// Rewrites the keys of the document `json` that a native key replaces.
+pub fn normalize<'j>(json: &'j [u8], options: &Options) -> Result<Normalized<'j>, Error> {


Because we are using serde it doesn't have to be json at this layer. Serde adds a level of normalization which we can use here. This would allow us to support yaml seamlessly just by adding the serde yaml dep

It would be great if we could use one parser that handles JSON and YAML, but I don't see viable open-source parsers to depend on. We can land this with JSON only for now, and discuss options for adding YAML support.

…out a file named normalize Review feedback on the README and the RFC. - What no native key says is now explained with the case it is about: a skip on a property is rewritten to `x-forge-skip` with the same value, but `x-forge-skip` is for every target, so a skip for only some targets, written as a list, has no native form and is left as it is written, in both modes. On an operation that key is not read at all. The RFC and the vendor README say it with the keys; the package README, which does not name other tools' keys, says it without them. - The README and `--help` no longer say how to pass a document that is itself called `normalize`.

Review feedback: there is no need to feature gate it. `serde_json` and `memchr` are dependencies, `forge::normalize` and `forge normalize` are always built, and the `normalize` feature, with the `cfg`s of the binary, the library, the native key tables and the test target that went with it, is gone. The `cli` feature still decides whether there is a `forge` command.

Review feedback: `forge` reads native keys only, so there is one key for each thing an extension says, and no need to remember which gave a value. `Val`, its `slot` and its `set` are gone, and the fields of a `Definition` are plain values. An error that names a key names the native key of the field, which `metadata::definition` now exposes, so the messages are the same. The released aggregate still writes the same 3,243 files of `--target ts`, byte for byte.

Review feedback: expanding a glob is a shell concern, and what to skip is the caller's to say. The inputs are files, or `-` for standard input to standard output. Folder expansion, glob expansion and the exclusion of hidden directories and `node_modules` are gone, and so is the `glob` dependency. A pattern the shell did not expand is a path that is not there, and a folder is an error that says so. A file named twice, or by two names of the same file, is still rewritten once.

Review feedback: a command that modifies files should say so. Without `--write`, `forge normalize` prints one document to standard output, rewritten, and leaves the file as it is; with several files it fails and says to use `--write` or `--check`. `--write` rewrites each file in place, only if something in it changes, as it did before. `--check` still writes nothing and fails if anything would change, and `-` still reads standard input and prints. `--write` cannot be given with `--check`, nor with `-`, which is not a file to rewrite. What it says when it prints is that the keys are "rewritten in the output", so that it does not sound like a file was changed.

Swimburger · 2026-10-28T17:18:09Z

TL;DR: forge normalize rewrites other tools' extensions to x-forge-*, editing only the keys it changes. That works for JSON. For YAML, the only parsers that can do the same are saphyr-parser and granit-parser, and I'm unsure whether we can depend on either. I'd like to know Cloudflare's dependency policy and how many teams have YAML specs.

Background: forge normalize rewrites x-fern-, x-stainless- and similar keys to x-forge-*. It edits only what's necessary, so a team migrating its own specs gets a clean diff: no whitespace changes, no lost comments, no number changes. It's built for our own specs first, and other Forge users can use it too.

The question is YAML support: Minimal edits need a parser that reports where each key and value sits in the source. I tested the Rust YAML parsers:
serde-based libraries lose comments and layout. On a 33-line sample spec, about 30 lines changed, every comment was dropped, and 1E5 became 100000.0.
yaml-rust2 has no end positions, and libyaml-rs is unsafe code translated from C.
That leaves saphyr-parser and its fork granit-parser. Both read JSON as YAML 1.2, so one parser could handle both formats.

	saphyr-parser	granit-parser
Age, size	2 years, 340 stars, 75 dependents	6 months, 12 stars, about 12 dependents
Maintenance	One active maintainer. The original author stepped back in January. Still 0.x, with four breaking releases in two weeks in October.	Mainly one maintainer. unsafe forbidden, fuzzed, monthly releases.
On our 28 MB aggregate	230 ms, 313 MB	180 ms, 29 MB

Both have small gaps on unusual valid JSON (for example saphyr-parser rejects \ud83d\ude00 escapes). Our own specs don't contain any of them, and we could fix them upstream if someone hits one.

Options I see (I'm not sure which fits):

JSON only for now. YAML teams convert to JSON first. No dependency decision is needed.
Depend on saphyr-parser, pinned, and send fixes upstream as issues come up.
Depend on granit-parser, pinned. Lighter, but the younger project.
Fork one of them into the Cloudflare org and depend on our fork.
Accept lossy YAML: reformat through a serde library, with a warning. This also costs memory and speed, because it holds the whole document as a tree. In my JSON test that took about 7× the file size in memory and 5× the time of our scanner. I haven't measured it on YAML.

My lean: if the policy allows it, option 2 or 3.

srikrsna · 2026-10-28T17:22:50Z

I don't think we need to add JSON and YAML parser just to preserve whitespace. We can simply read the json and yaml to guess the indentation they are at and output with the same indentation.

Swimburger · 2026-10-28T17:43:52Z

I ran a round trip with nothing renamed through serde_json (with preserve_order) and yaml_serde 0.10.7, on a 17,540-line slice of a real spec plus small crafted files. Likelihoods are my estimates unless marked measured.

Risk	Change	Likelihood
Silent, changes meaning	JSON duplicate key: the first copy is dropped, with no error	Low
	Numbers: decimals beyond about 17 digits are rounded, and integers above `u64` become floats in JSON (`100000000000000000001` becomes `1e+20`). YAML rounds too, with no option to prevent it.	Low
	YAML `1.10` becomes `1.1`, `0x1F` becomes `31`	Low to medium
Certain where present	YAML anchors and aliases are expanded into copies, and the reuse is lost	Medium
	YAML comments are all dropped	High
	YAML style changes: flow becomes block, `>` becomes `\|`, quotes, tags, `---` and blank lines	Certain
Loud error	Multi-document YAML, YAML integers above `u64`, a BOM or comments in JSON	Low to medium
Diff noise only	CRLF becomes LF on every line. Minified and inline JSON is expanded (3.6% of lines on prettier-style JSON, measured). `\u00e9` and `\/` are decoded.	Common

Guessing the indentation and writing it back works for files already in Python or serde layout (0 lines changed, measured). It doesn't help with anything above the last row, and yaml_serde has no indentation setting.

Cost: on a 27 MB JSON file, the round trip took about 5× the time and about 7× the file size in memory compared with the byte-level scanner. I haven't measured YAML.

A serde mode could refuse to write if the output differs from the input beyond the renamed keys, which turns the first group into errors. It can't bring back comments or style. The current approach edits only the renamed keys' bytes, so none of this applies to JSON.

Swimburger · 2026-10-28T21:34:15Z

Reviewers asked whether forge normalize needs its own JSON scanner, so I built the other option to compare: #75. It is the same change from the command line, but the document is read with serde_json and written again, in place of editing the bytes. Both PRs are open until we choose one, and the other should then be closed.

What #75 buys: about 900 fewer lines, no hand-written scanner, and a walk that is a table of what forge reads.

What it costs, on the 28 MB aggregate:

	#73 (bytes)	#75 (serde)
`forge normalize --check`	20 ms, 32 MB	90 ms, 184 MB
`forge --target ts`, document with old keys	260 ms, 159 MB	330 ms, 284 MB
`forge --target ts`, nothing to rewrite	240 ms, 153 MB	240 ms, 153 MB

What it gets wrong, and #73 does not. A file with a key to rewrite is not a minimal edit:

a key written twice loses its first copy, with no error (forge rejects it today)
numbers beyond 64 bits are rounded, and 1E5 and -0 are written 100000.0 and -0.0
arrays on one line are written on several (3.6% of the lines of a formatted document in a test), \r\n becomes \n, and escapes are written as the characters
an error position can be past the end of the file (forge says so)
JSON with comments is an error, and so is a document nested deeper than 128 levels

All of it is listed under "Known limitations" in the README of #75, and each item is pinned by a test.

What is the same. I ran --target ts, the cf CLI generator, and cf's own generate.ts pipeline on main, on this PR and on #75. The output is identical, down to the hash of the tree (7,226 files for cf). The one difference is that #75 keeps the old keys as ordinary extensions in memory, so plugins see four more than before.

The question that decides it: how many teams commit the files that forge normalize --write rewrites, and do they mind the diff? If specs are mostly generated or formatted by a tool, #75 is simpler and costs little. If teams edit them by hand, with their own layout and comments, #73 is the one that does not change anything it does not have to.

YAML: neither PR reads it, and both say so. A parser that reports where each value is, which editing YAML in place needs, has no dependency that I think meets the bar for this project, so that is separate from this choice.

This was referenced Oct 28, 2026

RFC 0001: metadata resolution layer (FORGE-1) #69

Open

feat: forge normalize, a command that rewrites other tools' extensions (FORGE-1) #72

Draft

Swimburger force-pushed the forge-native-keys-only branch from 491beec to d0ecf1f Compare October 28, 2026 01:01

Swimburger added 2 commits October 27, 2026 21:05

Swimburger force-pushed the forge-native-keys-only branch from d0ecf1f to fe43db8 Compare October 28, 2026 01:08

Swimburger marked this pull request as ready for review October 28, 2026 01:11

srikrsna reviewed Oct 28, 2026

View reviewed changes

Swimburger added 5 commits October 28, 2026 10:51

Swimburger mentioned this pull request Oct 28, 2026

feat!: forge reads native extension keys only, with forge normalize on serde (FORGE-1) #75

Open

Conversation

Swimburger commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Breaking

Performance

Behavior to look at

Verification

Follow-up

Uh oh!

srikrsna left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swimburger commented Oct 28, 2026

Uh oh!

srikrsna commented Oct 28, 2026

Uh oh!

Swimburger commented Oct 28, 2026

Uh oh!

Swimburger commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Swimburger commented Oct 28, 2026 •

edited

Loading