HTML API: Provide mechanism to scan all tokens in an HTML document, not only the tags.

[dmsnell]

Trac ticket: Core-60170
Companion port into Gutenberg: WordPress/gutenberg#58107 (contains additional porting code)

This PR provides full tokenization scanning of an HTML document. This is being added into the Tag Processor and will be a necessary component for a number of related changes to the HTML API:

• Reading and modifying inner and outer content.
• Serializing HTML.
• Methods to transform text content while preserving or stripping away markup.

Enables syntax-aware processing such as wp_truncate_html() [gist]
Replaces/incorporates chunked/extended processing in #5050
Replaces/incorporates stopping at comments in dmsnell#7
Provides critical functionality for inner/outer getter/setter in [dmsnell#10, #4965]

Depends on #5721 ✅
Depends on #5725 ✅

Todo

• Change CDATA sections and PI Nodes into comments with a new "comment type" flag.
• Ensure HTML Processor can seek around without messing up.
• Review all $this->bytes_already_parsed assignments and make sure they are proper. I think half of them are one off.
• Add function docblocks.
• Review what this enables in the html5lib test suite.
• Use this internally in the HTML Processor to ensure that breadcrumbs are generated.
• Explore using combinable bit flags for the token types instead of string constants. This would allow for things like MATCHED_TAG | TEXT_NODE and INCOMPLETE | COMPLETE, which could simplify some logic that's spread in if statements.
  • For this PR it's not worth moving to boolean logic like this. The existing code is clearer for review.
• Add test suite.
• Distinguish too-short HTML comments that may cause trouble when modifying. E.g. <!--->.
• Discuss what to do about PI Nodes and CDATA sections
  • It's possible to identify these after identifying the bogus comment span, but we can't look for the ending syntax of these sections because HTML stipulates that they end at the first >, not the closing ]]> or the closing ?>. So we can find all HTML comments, and then determine if they would have been a CDATA or PI Node if HTML supported those.
  • We can also ignore them all, but we lose knowledge about the HTML stream that we could recover (e.g. distinguish <?for-each?> from <--for-each-->.

Design Changes

In this change we're introducing two features stemming from two internal changes:

• next_token() provides the ability to scan every token in the HTML stream.
• it is possible to parse HTML in chunks without having the entire document in memory.

The internal changes powering this are:

• internal state adopts a new parsing mode which allows resuming from the middle of a previous match.
• the new concept of modifiable text and a token proper tracks the bounds of the currently-matched token as well as a safe region that can be changed without impacting the document syntax, if one exists.

For example, when encountering an HTML comment the parser will track the following token information:

This <!-- is a comment -->.
     │   │            │  └ end of token
     │   │            └─── end of text
     │   └──────────────── start of text
     └──────────────────── start of token

Not every token will have a text region, but it's important to track the entire token and any text region because similar tokens may have different syntax. For example, an invalid comment is still a comment.

This <? is also a comment --!>.
     │ │                 │   └ end of token
     │ │                 └──── end of text
     │ └────────────────────── start of text
     └──────────────────────── start of token

This holds for tokens whose entire content is text, such as with the #text node.

<div>This is text.</div>
     │           ├ end of token
     │           └ end of text
     ├──────────── start of text
     └──────────── start of token

Special HTML tags have modifiable text and that isn't part of .textConent or .innerText. For example, the TITLE element contains no HTML inside of it and everything is plaintext and its contents don't appear in the page. The same is true for TEXTAREA and SCRIPT and STYLE and a few more elements.

<title>This is text <em>Not HTML</em>.</title>
│      │                             │       └ end of token
│      └ start of text               └ end of text
└──────────── start of token

Scanning tokens

In order to keep the next_tag() interface and use clear, it is left unchanged. For operations needing access to the token stream, there is no built-in query mechanism and querying ought to be performed inside a next_token() loop. get_token_type() indicates what kind of token is currently matched, get_token_name() returns something that more closely matches what a DOM API would return, and get_modifiable_text() returns the modifiable text if available.

function wp_strip_all_tags( $html, $remove_breaks ) {
	$processor = new WP_HTML_Processor( $html );

	$text_content = '';
	while ( $processor->next_token() ) {
		if ( '#text' === $processor->get_token_name() ) {
			$text_content .= $processor->get_node_text();
		}
	}

	return $remove_breaks
		? trim( preg_replace( '/[\r\n\t ]+/', ' ', $text_content ) )
		: $text_content;
}

• Most tags have no modifiable content.
• The inner contents of special tags whose contents cannot contain HTML is their modifiable content. The inner contents of these tags are not rendered in the page.
  • IFRAME
  • NOEMBED, NOFRAMES
  • SCRIPT
  • STYLE
  • TEXTAREA [character references are decoded]
  • TITLE [character references are decoded]
  • XMP

TODO

• Add next_token() method to scan each token.
• Stop at RCDATA sections and SCRIPT, STYLE, TITLE, TEXTAREA, etc…
• Stop at text nodes.
• Indicate a continuation state to support resumable parsing. This is necessary for stopping at SCRIPT tags and other tags with special closing rules. These are currently handled by skipping to the end of the element when finding the starting tag, but this has introduced a few challenges and bugs (for example, the Tag Processor fails to stop at a <title> tag if the document ends before the </title> closer is found).
• Add rewind() method to reverse to the start of the document.