Write the PHPDoc language down as a grammar, and fuzz from it

.gitattributes

@@ -1,16 +1,16 @@
-*.abnf text eol=crlf
 *.php text eol=lf
+*.pp3 text eol=lf
 
 .github export-ignore
 apigen export-ignore
 phpcs.xml export-ignore
 doc export-ignore
 tests export-ignore
+tools export-ignore
 tmp export-ignore
 .editorconfig export-ignore
 .gitattributes export-ignore
 .gitignore export-ignore
-build-abnfgen.sh export-ignore
 CLAUDE.md export-ignore
 CODE_OF_CONDUCT.md export-ignore
 Makefile export-ignore

.github/workflows/build.yml

@@ -140,6 +140,44 @@
       - name: "Tests"
         run: "make tests"
 
+  grammars:
+    name: "Grammars"
+    runs-on: "ubuntu-latest"
+
+    strategy:
+      fail-fast: false
+      matrix:
+        php-version:
+          - "8.4"
+          - "8.5"
+
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
+        with:
+          egress-policy: audit
+
+      - name: "Checkout"
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: "Install PHP"
+        uses: "shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240" # v2
+        with:
+          coverage: "none"
+          php-version: "${{ matrix.php-version }}"
+          tools: composer:v2
+
+      - name: "Install dependencies"
+        run: "composer update --no-interaction --no-progress"
+
+      # The compiler reading doc/grammars asks for PHP 8.4, so the corpus is
+      # only written here. Everywhere else FuzzyTest skips itself.
+      - name: "Install the grammar toolchain"
+        run: "make grammars-install"
+
+      - name: "Tests"
+        run: "make tests"
+
   static-analysis:
     name: "PHPStan"
     runs-on: "ubuntu-latest"

.gitignore

@@ -1,6 +1,6 @@
 /docs
 /temp
-/tools
+/tools/phplrt/vendor
 /tests/tmp
 /build-cs
 /vendor

Makefile

@@ -34,3 +34,14 @@ phpstan:
 .PHONY: phpstan-generate-baseline
 phpstan-generate-baseline:
 	php vendor/bin/phpstan --generate-baseline
+
+# ---------------------------------------------------------------------------
+#  The grammars
+# ---------------------------------------------------------------------------
+
+# The tool writing a corpus out of doc/grammars is a development dependency of
+# its own, because the grammar compiler asks for a PHP this library still runs
+# without.
+.PHONY: grammars-install
+grammars-install:
+	composer install --no-interaction --working-dir tools/phplrt

README.md

@@ -166,6 +166,22 @@ $newPhpDoc = $printer->printFormatPreserving($newPhpDocNode, $phpDocNode, $token
 echo $newPhpDoc; // '/** @param Ipsum $a */'
 ```
 
+## The grammars
+
+The language this library reads is written down as a grammar in
+[`doc/grammars`](doc/grammars), in the format the [phplrt](https://phplrt.org)
+compiler reads. A grammar says what a PHPDoc may be written as, so it can be
+walked the other way round and asked for PHPDocs instead of being asked about
+one: that is where `FuzzyTest` gets its corpus, and it covers a great deal more
+of the language than a hand-written one does. `GrammarSyncTest` walks them the
+other way, asking each grammar about the inputs of every other test in this
+project, so that what the grammars describe and what the parser reads cannot
+drift apart.
+
+Nothing in `src/` reads those files, and the library needs neither the toolchain
+writing the corpus nor the PHP 8.4 it asks for. See
+[`doc/grammars/README.md`](doc/grammars/README.md).
+
 ## Code of Conduct
 
 This project adheres to a [Contributor Code of Conduct](CODE_OF_CONDUCT.md). By participating in this project and its community, you are expected to uphold this code.
@@ -181,3 +197,13 @@ Afterwards you can either run the whole build including linting and coding stand
 or run only tests using
 
     make tests
+
+The grammars have a toolchain of their own, because the compiler reading them
+asks for PHP 8.4. Without it the fuzzy tests skip themselves:
+
+    make grammars-install   # install it
+
+`FuzzyTest` then writes its own corpus out of `doc/grammars/*.pp3` every time it
+runs, and leaves it in `temp/fuzzy` to be looked at afterwards. `GrammarSyncTest`
+asks the same grammars about the inputs of every other test, so that a feature
+added to the parser is one they have to describe as well.

doc/grammars/README.md

@@ -0,0 +1,153 @@
+# The grammars of the PHPDoc language
+
+The files in this directory describe the language `phpstan/phpdoc-parser` reads,
+in the [PP3 format](https://phplrt.org/docs/basics/grammar) the
+[phplrt](https://phplrt.org) compiler reads. They are the specification of the
+language, and they are what `tests/PHPStan/Parser/FuzzyTest.php` writes its
+corpus from.
+
+| File                | What it holds                                                   |
+|---------------------|-----------------------------------------------------------------|
+| `lexemes.pp3`       | Every token the language is read into                           |
+| `common.pp3`        | What the grammars share: names, brackets, line breaks           |
+| `types.pp3`         | The type language of `TypeParser`                               |
+| `const-expr.pp3`    | The constant expressions of `ConstExprParser`                   |
+| `phpdoc-block.pp3`  | The PHPDoc itself: its tags, its text, its Doctrine annotations |
+| `type.pp3`          | The entry point starting at `Type`                              |
+| `constant-expr.pp3` | The entry point starting at `ConstantExpr`                       |
+| `phpdoc.pp3`        | The entry point starting at `PhpDoc`                             |
+
+## What they are for
+
+A grammar says what a PHPDoc may be written as, so it can be walked the other
+way round and asked for PHPDocs instead of being asked about one:
+
+    make grammars-install                  # the toolchain, which asks for PHP 8.4
+    php vendor/bin/phpunit --filter FuzzyTest
+
+`FuzzyTest` runs `tools/phplrt/fuzz.php` itself, once per grammar, every time it
+runs: the tool compiles a grammar, walks its rules at random and writes down
+what comes out, and the test then asks the parser to read every one of them in
+full, and to read a type back as the very same type once it has been printed.
+There is no corpus to keep, and none to refresh when a rule changes — what is
+generated changes the moment the grammar is read again. The inputs of the last
+run are left in `temp/fuzzy` to be looked at.
+
+That is a great deal more of the language than a hand-written corpus covers, and
+it is what replaced the `abnfgen`-driven fuzzer this project used before.
+
+## What keeps them in step
+
+The fuzzer walks the grammars one way; `tests/PHPStan/Parser/GrammarSyncTest.php`
+walks them the other. It reads the inputs of every other test in this project —
+by reflection over the providers they are written with, so the corpus grows on
+its own as they do — and asks each grammar about them through
+`tools/phplrt/recognize.php`:
+
+- everything the parser reads in full and without complaint, the grammar has to
+  recognize;
+- everything the parser stops at or turns into an `Invalid...` node, the grammar
+  has to turn down.
+
+So a feature added to the parser and written down as a test is one the grammars
+have to describe by the next run, and a rule written too widely is one the
+fuzzer walks into. Only reading is asked of the grammar there: nothing is built
+out of what it reads, so there is no second implementation of the AST to keep in
+step — only the language itself.
+
+The few inputs the two are known to disagree about are listed in that test with
+the reason why, and the list has to be exactly right: an input that starts
+agreeing has to be taken off it.
+
+Nothing in `src/` reads these files, and the library needs neither the toolchain
+nor PHP 8.4: where either is missing, `FuzzyTest` skips itself. Only the
+`Grammars` job of `.github/workflows/build.yml` runs it for real.
+
+## How they are written
+
+The rules are named after the methods of `PhpDocParser`, `TypeParser` and
+`ConstExprParser` they stand for and are written in the order those methods try
+things in, so that a grammar and the parser it describes can be read side by
+side.
+
+### What the grammars describe
+
+**A PHPDoc that is written correctly.** The parser reads a broken one as well,
+by turning whatever it cannot read into an `InvalidTagValueNode` carrying the
+very error it has raised, and a grammar has no way of writing that error down.
+So what a broken PHPDoc means is left to the parser, and everything the grammars
+describe is something the parser has to read in full.
+
+Two things follow from wanting that to hold for **every** input rather than for
+most of them:
+
+- **A place the parser raises an error at is written as something the grammar
+  cannot recognize.** Most of them are written as a `!` predicate forbidding
+  whatever the error would have been raised on. For instance a name followed by
+  a `<` has to go on into a generic type or into a callable, because `Foo<` is
+  an error rather than the type `Foo` followed by something else:
+
+  ```
+  IdentifierAtomic
+    : ...
+    | !ShapeBrace() Identifier() !<T_DOUBLE_COLON> ( IdentifierSuffix() | !<T_OPEN_ANGLE_BRACKET> )
+    ;
+  ```
+
+  The same predicate is what keeps a rule from **giving back** what it has read.
+  `@template T of` is an error rather than a template named `T` with the
+  description `of`, so the bound is written as "either a bound or no `of` at
+  all":
+
+  ```
+  TemplateUpperBound
+    : <T_KEYWORD_OF> Type()
+    | <T_KEYWORD_AS> Type()
+    | !<T_KEYWORD_OF> !<T_KEYWORD_AS>
+    ;
+  ```
+
+- **A rule reads exactly the tokens its method reads**, down to the line breaks
+  around it.
+
+### The tokens are not read by phplrt
+
+A grammar of this directory is not read by the lexer it declares: it is read by
+the very tokens `PHPStan\PhpDocParser\Lexer\Lexer` produces, handed over by
+`tools/phplrt/Fuzzer/TokenStream.php`.
+
+The `%token` declarations therefore name the tokens and document the language
+without being what reads it. Some of them describe something the lexer never
+reads as a token of its own, and `TokenStream` is what tells those apart in the
+stream:
+
+- a word the parser compares by value (`is`, `array`, `covariant`, `static`, …)
+  — every one of them is still an ordinary name as well, which is why they are
+  all listed among the alternatives of `Identifier`;
+- a tag whose value a rule of its own reads (`@param`, `@return`, …), told apart
+  from the tags nothing reads the value of;
+- a bracket or an asterisk whose neighbouring whitespace decides what it means,
+  which is what tells `array{a: int}` from the type `array` followed by a brace,
+  and `Foo[0]` from `Foo [0]`;
+- a tag a space is written before, which is what tells the `@since` of
+  `@author Foo @since 1.0` from the `@baz` of `@author Foo <foo@baz.com>`;
+- a `<` opening what the parser recognizes as an HTML tag, so that
+  `@return Foo<br>see below</br>` keeps meaning the type `Foo` followed by a
+  description.
+
+Telling them apart there is what lets the grammars be written without semantic
+predicates, which the PP3 format has none of.
+
+### What is left out
+
+Three corners of the language are left out on purpose, because a grammar cannot
+say what the parser does there. Each of them is written up where the rule that
+skirts it is written:
+
+- a description that ends at a tag written in the middle of a line, which the
+  parser decides by reading the tag and looking at what it turns out to be;
+- the same, on a line after the first, where the parser reads that line twice:
+  once as part of the description and again as whatever comes next;
+- a tag whose value a rule reads, written with a parenthesis after it, where
+  whether the description ends there depends on whether that value can be read
+  at all.