feat: add max/min count thresholds to sequence rules - #1162
Open
theredspoon wants to merge 3 commits into
Open
Conversation
theredspoon
marked this pull request as ready for review
September 2, 2026 01:02
This was referenced Sep 2, 2026
theredspoon
force-pushed
the
feat/sequence-max-min
branch
2 times, most recently
from
September 2, 2026 02:42
3f86ddb to
5fb16d5
Compare
theredspoon
force-pushed
the
feat/sequence-max-min
branch
2 times, most recently
from
September 2, 2026 03:06
839eab5 to
86405ce
Compare
The remote-tagging and remote-segmentation call sites (TextToTokens,
Info.Compute) panicked on a failed /tag or /segment request instead of
returning an error, crashing the whole vale process rather than
surfacing a normal, reportable error. TextToContext and the `tag` CLI
command are updated to thread the error through rather than let it
panic.
The shared HTTP transport also ignored response status codes: a non-2xx
response with a technically-valid JSON body (e.g. `500 {"sents":[]}`)
was silently decoded as a successful, empty result instead of a failure.
Each fix has its own regression test.
Two internal/e2e scenarios cover the user-visible behavior end to end,
both against a closed local port so the failure (connection refused) is
deterministic and needs no network or mock server: a lint run whose
Info.Compute hits a failed /segment request during block construction,
and the `tag` CLI command's /tag request. Verified against the pre-fix
commit that both currently fail this way (a panic with a goroutine
stack trace) before this fix, and pass cleanly after it.
sentenceScope's negation branch was a no-op for a bare negated term: `~list` narrowed to `~list`, itself, via strings.CutPrefix re-adding the same prefix it had just stripped. A negated term never mentions `sentence`, so asksForSentence (scope.go) then skipped every `sentence.*` fragment block for such a rule, and Scope.Matches instead matched both the whole-block copy and its own paragraph wrapper for the same text. One real match dispatched to Run twice, once per block, and produced two identical alerts. The negation branch now AND-s `sentence` in front of the term instead of leaving it untouched, so `~list` narrows to `sentence&~list`, sentences outside a list, the same as every other declared scope already does.
theredspoon
force-pushed
the
feat/sequence-max-min
branch
from
September 2, 2026 03:38
86405ce to
2643c6d
Compare
NewSequence unconditionally narrowed every declared scope to sentence-level, so a rule using max/min could never aggregate matches across a paragraph's sentences. Threshold-opted-in rules now keep their real declared scope; Run tags each sentence of that scope separately instead of tagging the whole block once and inferring sentence boundaries afterward, so a match can never span two sentences by construction. An undeclared scope on a threshold rule now defaults to paragraph plus every other prose-container scope, matching what a plain sequence rule's undeclared scope already reaches, via one shared list in internal/core instead of two independently-maintained copies. Built on vale-cli#1167 (fixes three pre-existing panics on remote NLP endpoint failures this feature's own paths would otherwise have hit) and vale-cli#1169 (fixes a sentenceScope bug that review of this feature found as a real, dispatched double-report).
theredspoon
force-pushed
the
feat/sequence-max-min
branch
from
September 2, 2026 03:49
2643c6d to
99f0ca0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1161
Review note: this branch is rebased on #1167 and #1169 (both needed underneath for correctness, see Fix). Only the commit
fix(sequence): count-threshold rules dispatch at declared scope, tagging each sentence separatelybelongs to this PR. Click into it on the Commits tab to see this PR's actual diff. The Files changed tab includes #1167's and #1169's content too, and will shrink once both merge intov3and this rebases.Problem
sequencematches tagged patterns but only alerts once per match. It has no count threshold.occurrencehasMax/Minbut only matches a raw regex, notag/upos. Root cause:NewSequenceunconditionally narrowed every declared scope to sentence-level, so a paragraph with two real matches split across two sentences stayed silent, since each sentence individually had only one match.tbhb/vale-ai-tells'sVerbTricolonDensity.ymlneeds exactly this. Itsoccurrenceregex can't require the matched words be verbs. Asequencerule can, and now finally gets a count threshold too.Fix
Runnow tags each sentence of a rule's real declared scope separately (reusinginternal/nlp/prose.go's existing per-sentence tagging), instead of tagging the whole block once and inferring boundaries afterward. Sentence membership becomes a direct fact, so a match can't span two sentences by construction.Max/Minset) keep their real declared scope instead of being narrowed to sentence-level.File.Sentencesroutes through the same local/remote segmentation dispatch the rest of the codebase already uses.Why the redesign, not a smaller patch (root cause detail)
Sentence boundaries used to be inferred after tagging a whole block once: compare each word's offset against each sentence's own offset. That broke down for a remote endpoint's unpositioned tokens, which carry no offsets to compare. Tagging each sentence separately makes sentence membership a direct fact instead, determined by which loop iteration produced the word.
Performance
Local English tagging: neutral, within noise. Non-English remote endpoint: a plain rule now pays up to 2 round-trips per sentence instead of 1. A separate follow-up, #1170, removes the remaining redundant one for plain rules. Match-walk cost drops from one O(N²) per block to a sum of smaller O(N²/k) terms across k sentences.
Full performance reasoning
For a style's
sequencerules as a whole, this is close to neutral for local English tagging. The new per-sentence segmentation pass is cached per file, so only the first rule to touch a sentence pays for it. Every other rule sharing that file's cache gets a hit. One rule against a cold cache measures a few percent slower. Benchmarked with several rules sharing a file's cache, the difference is within noise and can go either way.sequenceMatches' match-walk is quadratic in the number of words per call. Splitting a block into sentences turns one large quadratic term into a sum of smaller ones,N²/kinstead ofN²forksentences. Each sentence also pays a small, fixed per-call cost: tokenization and a fresh lookup map. A block of many very short sentences can let that fixed cost outweigh the quadratic saving. Realistic prose doesn't hit this.Sentence segmentation is cached: a repeated call against the same text hits a map lookup, measured at roughly 140x faster than a fresh segmentation.
A non-English document behind a remote NLP endpoint pays an extra HTTP round-trip, but not a flat doubling. A plain (sentence-scoped) rule now makes one segmentation call and one tagging call per sentence, two round-trips where there was one before. A
max/minrule segments its whole scope in one call, then tags each sentence individually: one extra segmentation round-trip per block rather than per sentence, shrinking as a fraction of the total as a block's sentence count grows. Both calls are cached per file, so a style with manysequencerules pays this once per unique sentence, not once per rule. Local English tagging, the common case, never touches this path at all.Testing
internal/e2ecase (checks/sequence/max) for the density-across-sentences motivating case, end-to-end through a real.vale.ini/style/documentFull repo suite and
-raceare clean, includinginternal/e2e. Confirmed no reintroduction of the concurrency risksequencewas historically excluded from concurrent dispatch for.Alternative considered: extending occurrence.go instead (rejected)
occurrenceruns concurrently across rules matching one block.sequenceis deliberately excluded from concurrent dispatch because its tagging cache isn't synchronized. Fixable, but it touches shared dispatch infrastructure other check types rely on and givesOccurrencetwo structurally different matching modes.Related
min: Nper-token repetition (#899) ·scope: paragraphfix (#1124/#1126) · remote NLP endpoint panic fixes, split out to #1167 ·sentenceScopenegated-scope double-report fix, split out to #1169 · redundant round-trip elimination, follow-up in #1170