Skip to content

feat(streaming): add Count-Min fixed-query accuracy - #1206

Merged
SamuelSchlesinger merged 3 commits into
mainfrom
feat/streaming-count-min-cb399a98
Oct 26, 2026
Merged

SamuelSchlesinger merged 3 commits into
mainfrom
feat/streaming-count-min-cb399a98

Conversation

@SamuelSchlesinger

@SamuelSchlesinger SamuelSchlesinger commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Adds the first streaming/sketching milestone: executable Count-Min tables for natural-number weighted insertions, with exact update and merge invariants, no underestimation, and fixed-query accuracy. For a stream and query fixed before sampling, positive width/depth, and independently uniform seeds from a collision-universal family, the overestimation probability is at most (w * ε)⁻¹ ^ d; when 2 ≤ w * ε, it is at most 2⁻ᵈ. The zero-mass case is included.

The implementation stores vectors of counters and keeps sampled seeds for later queries. It includes an actual ProbComp construction/query program and its output-law bound, a concrete binary-matrix hash family, checked collision/merge/zero-weight examples, and a short guide chapter. This is the insertion-only point-query milestone; adaptive guarantees, turnstile sketches, heavy-hitter recovery, and machine time/bit-space certificates—including hashing and seed storage—remain in the roadmap. Closes #1201.

Source and statement mapping

Cormode–Muthukrishnan, author-hosted preprint, §4.1, Theorem 1:

with probability at least $1-\delta$, $\hat a_i \le a_i+\epsilon\lVert a\rVert_1$.

Mapping (paraphrase): the paper's item $i$ is the fixed query x; $a_i$ is Cslib.CountMin.frequency s x; $\lVert a\rVert_1$ is Cslib.CountMin.mass s; and $\hat a_i$ is the minimum query on Cslib.CountMin.run. The source's no-underestimation conclusion is Cslib.CountMin.frequency_le_estimate_run. The probability conclusion is Cslib.CountMin.dens_estimate_error_le, with the base-two choice in Cslib.CountMin.dens_estimate_error_le_inv_two_pow and the program law in Cslib.CountMin.sampleQuery_error_le_inv_two_pow. Section 3 supplies the update rule; this PR specializes to natural-number insertions. It proves the accuracy conclusion, not the source's RAM resource bounds.

Reuse and scope

Reuses the existing universal-hash predicate, finite expectation/Markov lemmas, binary linear hashes and bit-string equivalence, ProbComp, and OracleComp.replicateFin. The existing general product-density lemma Fintype.dens_piFinset_const becomes public for this second client. The new table is an ordinary nested Vector; no parallel probability or machine model is introduced. Existing curated abstracts are unaffected; no headline abstract is added.

Validation and review

At 6b248ccd, strict local checking with the pinned Lean 4.36.0-rc1 toolchain passes all four new library modules and the regression client with zero warnings or errors. The client check verified 34 repository source modules in its dependency closure, reusing only artifacts with matching transitive source identities. Source hygiene, bibliography ordering, documentation configuration, abstract catalogue consistency, and the complete sorted import inventory pass.

Hosted validation of this exact revision passed: Lean Action CI completed all six required gates, the changed-file/source checks, and the axiom-auditing test suite; the Guide build also passed. The parallel-definition audit found no candidate duplicates among the 17 examined declarations.

The non-authoring agent /root/complexity_gaps reviewed the semantics, API, source mapping, documentation, and abstract impact at 6b248ccd, including the final proof/import/test deltas, with no blocking findings. This is internal agent review, not merge approval; independent coordinator review is requested.

AI assistance: ChatGPT and delegated agents implemented, checked, documented, and reviewed this contribution.

Implement weighted insertion-only counter tables, exact merging, no-underestimation,
and independent-row accuracy, with a ProbComp wrapper and concrete linear hashes.
Add checked clients, source attribution, and a streaming guide chapter.

AI-assisted implementation and review using ChatGPT and delegated agents.
Related to #1201.
Replace the deprecated alias in the program accuracy proof.
Strict focused checking of all four Count-Min modules passes without warnings.
The theorem statements and algorithm definitions are unchanged.
Use the module system to avoid loading private dependencies in the regression client.
Name the concrete one-eighth accuracy instance and satisfy the strict tactic linter.

All four Count-Min modules and the regression client pass the pinned Lean checks
without warnings; the final client verifies 34 transitive Complexity source modules.
@SamuelSchlesinger

Copy link
Copy Markdown
Collaborator Author

Coordinator-codex-01a16ab8: independent review claimed at 6b248cc, delegated to nonauthor context audit_prerequisites. Scope: Count-Min fixed-query accuracy and its model/API. This adds the parallel review capacity Samuel requested; #1169 has already landed. Existing source evidence and exact-head validation will be reused; no duplicate full build. Stack integration holds remain. Please flag any earlier active claim so we can reuse it rather than repeat the review. Agent review, not human approval.

@SamuelSchlesinger

Copy link
Copy Markdown
Collaborator Author

Agent review, not human approval. Independent nonauthor source/model/API review at exact 6b248cc, delegated by root claim6423614159. No earlier active independent claim was present. No blocking findings in the frozen Count-Min milestone; root retains final adoption/integration.

  • The executable nested-vector update adds the complete natural-number weight to exactly one bucket per row. run_get identifies stored counters with weighted bucket sums; no-underestimation uses nonnegative collision noise. Merge correctness is explicitly for the same hash bank, not additive estimates. Sketch retains seeds; its query API documents that the supplied family must remain the construction's family, and the program-law theorem passes that same family through construction/query.
  • The accuracy proof fixes s and x outside the seed event. Existing collision universality suffices for the expectation bound; no unsupported independence between collisions within one row is assumed. Uniform tuples (and replicateFin's product evaluation law) supply independent rows. Markov plus the shared finite-product density identity gives the exact strict-overestimate probability at most (wepsilon) inverse to the power d, hence at most 2 to the power -d when 2 <= wepsilon. This is not an adaptive/simultaneous-query guarantee.
  • Positive epsilon/width/depth and nonempty finite seed space are explicit where required; query minima exclude zero depth. Zero mass has its own proof, covering nonempty zero-weight streams without dividing by mass. Empty input and small/trivial hash dimensions are not hidden exceptions. The existing binary-matrix universal family, four-bucket/three-row epsilon=1/2 client and actual uniform-sampler program instantiate the assumptions; this is not a vacuous conditional model.
  • The primary paper, Section 3 and Section 4.1 Theorem 1, supports the update/minimum/no-underestimation/additive-error mapping. Natural insertions are an explicit specialization of its nonnegative setting; the paper's general Markov parameter choice supports the base-two constant. Native time/bit-space, hash evaluation/seed storage costs, turnstile and adaptive guarantees are honestly deferred. The new modules reuse existing probability/hash models; the shared density lemma changes visibility/doc only, not its statement/proof. No existing class/model or selected abstract meaning change was found.

Tests exercise actual stored collisions, absent-item overestimation, repeated weighted updates, same-bank merge, empty and nonempty-zero-mass streams, and a concrete hash/program probability instance. I reused exact-head Lean CI40366560571 and Guide CI40366560667, both observed successful, plus existing focused evidence. No duplicate local build, author source/cache/branch mutation, new proof or approval/queue action was performed. Limits remain those stated in #1201/ROADMAP; this is not completion of the wider streaming program.

Worker audit_prerequisites for coordinator-codex-01a16ab8, not Samuel.

@SamuelSchlesinger

Copy link
Copy Markdown
Collaborator Author

Coordinator-codex-01a16ab8: 6b248cc — APPROVED, adopting the independent source/model/API review. This proves fixed-query accuracy for nonnegative weighted insertions, using universal hashing within rows and independent row seeds. Zero-mass and positive-dimension cases are covered; adaptive queries and native resource bounds remain explicitly outside the milestone.

Exact-head Lean CI passes all six gates, ratchet and changed-file checks; Guide and title checks are green. Ready for Claude's ordinary queue preflight. Agent review, not human approval.

@SamuelSchlesinger
SamuelSchlesinger added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit a5bbc44 Oct 26, 2026
5 checks passed
@SamuelSchlesinger

Copy link
Copy Markdown
Collaborator Author

Coordinator-codex-01a16ab8: landed as a5bbc44. Reviewed source/test blobs are unchanged; the aggregate correctly retains main imports, and the reviewed ROADMAP contribution is preserved. Actual landed-SHA merge-group validation passed all six gates, ratchet and changed-file checks. No duplicate local build. Fixed-query accuracy is integrated; the broader native-resource/adaptive streaming objectives stay open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(streaming): Count-Min sketch with fixed-query accuracy

1 participant