Skip to content

feat(wordram): complete native Chan–He preprocessing - #1537

Closed
edemaine wants to merge 104 commits into
mainfrom
t3/native-chan-he-preprocessing-forest
Closed

edemaine wants to merge 104 commits into
mainfrom
t3/native-chan-he-preprocessing-forest

Conversation

@edemaine

@edemaine edemaine commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Completes the native preprocessing implementation assigned in #1446. Targets main and must merge after #1535; its refreshed traversal and frontend/node suppliers are included by ordinary merges. One finite 128-register program per fixed κ starts from actual native State.init of the signed-encoded input, computes every parameter, sorts/deduplicates with full multiplicities, enumerates every distinct-position frontend root, builds the prime bitmap, executes both modulus searches and all heavy/light recursion, and writes exactly the existing ChanHe.instances in order. #1394 remains open: the native Exact Triangle solver and complete final 3SUM composition, including the improved route to exponent 1.9992, are separate work.

Current documentation restoration, landed-main refresh, validation and review: integration record. All 31 own source/test files retain their reviewed versions, and the explicit resource contracts and bounded API handoff are restored. Actual #1535 remains a prerequisite; the full #1394 endpoint remains open.

The API is in ChanHe/Preprocessing.lean. Preprocessing.runs_input provides native halting execution and the same actual pre-halt memory witness; exec_input supplies structured execution. For every κ, every input of length n with |x|≤(n+2)^κ, and every physical width w≥(1300κ+7865)⌈log₂(n+2)⌉, output count is register 79, pointer-table base 78 and common length 116. Metadata is unsigned; pointed-to cells use the existing signed integer encoding. Each array has the exact existing length 8m², where m=mPar n (2(n+2)^κ). The actual input-length parameter stays n, including n=0,1,2.

Preprocessing.cost_bound proves Kκ·(n+2)·(√(n+2)+1)·(log₂(n+2)+1)^13 charged instructions, including the final halt, with explicit Kκ independent of n, data and admitted width. This explicit square-root-times-linear fixed-polylog bound uses original native trial division and histogram clearing. The pinned source's TimeBound.lean, lines 20–24, also describes n^(3/2) times a polylogarithm for the host's own work; the charged bound here is proved directly over native WordRAM instructions. Every arithmetic/address operation, descriptor load, stack action, loop, prime search, heavy extraction and array write is charged; no pair-count estimate substitutes for runtime. arena_ready derives all local fits from positive global logarithmic width. intermediates_polynomial and space_polynomial cover every prefix at every larger admitted width: all words and the occupied initial address range are bounded by (n+2)^(2720κ+16456).

Exact equality with the existing finite list preserves its degree-seven logarithmic count, linear-times-fourth-logarithm common length and 120(n+2)^κ+40 value bound; empty input emits no arrays. The admitted negative two-position client [1,1] proves nonempty exact output and absence of a convolution solution. Other clients cover lengths 0 and 1 and the equal-value positive input [-2,1,1], retaining actual execution/memory witnesses. Interpreter regressions execute the complete empty-input pipeline and a genuine width fault.

Source mapping: checked at anthropics/formal-math@16a5ce4, ThreeSumApsp/Programs/Sec3/Theorem21a/ChanHe/Program.lean, lines 23–24:

The procedures of the reduction are appended to the program of an arbitrary solver of
Convolution-3SUM.

That source assembles a solver-calling Light host. This PR implements preprocessing as original native code enumerating its existing finite instances. It supplies no solver and imports no Light program, machine, runtime oracle or simulation assumption. Reused finite mathematics retains Anthropic, PBC attribution. Native controller/layout/execution/resource proofs are original. The shifted polynomial magnitude convention admits empty input and exponent zero. Source notation maps to existing nodes/instances, candidate window, labelBits, minOrOne and signed encoding; actual WordRAM.Code/ExecWithin and native compilation supply the execution model.

Reuse/API: existing finite reduction, frontend/parameters, radix/multiplicity, prime, modulus, node, block-memory and compiler APIs are composed without parallel computational notions. A small frontend frame lemma serves the forest. The permitted #1527 follow-up is completed: private width-renaming/prefix lemmas move into the small generic WordRAM.Width topic module and serve both clients; its input-module result list and redundant predicate-only tests are cleaned. The #1535 traversal-doc nit is fixed here. Ancestor implementation/review ownership is preserved; no ancestor remotes change. The generic signed-fit/count placement follow-up is implemented separately in #1552. This bounded outside-review repair corrects the stale UPSTREAM remaining-work sentence, shares private parameter-scale bounds, removes the unused cube estimate, clarifies the umbrella re-exports and traversal parameter/cost documentation, and proves that the branching mathematical task produces the same two nodes consumed by the existing native interpreter check. UPSTREAM records concrete stage-plumbing, selector-placement and loop-rule reuse follow-ups without changing those public interfaces.

Validation at a33aea0 (tree e22936eadaf3760f756ef40db4b27c057b9f5341): all six strict gates passed together, including global Axioms; actual-base changed-file lints, full hygiene/simp, strict PR-name and diff audits passed. Curated abstracts: no impact; public theorem statements, native programs, models, encodings and resource bounds are unchanged. The prior nonauthoring technical review and outside full review cover the unchanged implementation; focused outside delta review is requested for the repair. This component is not the final subquadratic 3SUM headline. AI assistance: OpenAI Codex implemented and checked code, proofs, clients and documentation. Agent review is not human approval.

Adapt the finite counting interface and signed digit expansion from anthropics/formal-math@16a5ce4 for milestone #1438. This is a mathematical interface; native preprocessing, fast solvers and full composition remain under #1394.
Adapt the complete finite Theorem 17 splitting and recovery component, using the existing three weight matrices, lopsided graphs, modular counts and EqOn answer interface. Prove exact integer/real capacity bridges so the finite constructions and scans are computable. Preserve Anthropic source credit; keep the native 3SUM endpoint open in #1394.

Validation: all six gates in scripts/check.sh pass, with LEAN_NUM_THREADS=1 and cache downloads disabled after bootstrap. Full hygiene, changed-source check-simp, changed-file lints and git diff --check pass. Curated abstracts and rendering are unaffected. AI-assisted implementation/self-review by Codex worker codex-three-sum-splitting-t3-0a4d90c0; independent review will be recorded in the PR.
Address the nonblocking review finding on #1455: finite residue/chunk/piece splitting and witness recovery are implemented. Preserve native preprocessing, fast solvers, full input/time/width composition and exponent 1.9992 as open work under #1394.

Documentation only. Lean and test files are byte-identical to independently reviewed e41e1dd, whose six strict gates pass. Checked the diff, milestone reference, implementation paths, hygiene and changed-file lints. Abstracts and rendering are unaffected.
Implement fixed native residue, histogram and composed collision-counting
programs with explicit frames, local word-fit contracts, exact mathematical
collision semantics and charged bound 34n + 5M + 18 (halt excluded).

Implementation layer for #1454 under #1446/#1394, not full preprocessing.
Sorting/frontend, prime searches, heavy extraction, recursion/output and
global logarithmic-width/subquadratic-time composition remain open.
Curated abstracts: no impact on headline statements, definitions or scope.

Validated on base 8e182b0 plus this exact diff:
LEAN_NUM_THREADS=1 COMPLEXITYLIB_CACHE_AUTO=0 scripts/check.sh: all six gates
passed, including strict test/global axiom audit. Changed-file lints against
that base, hygiene --check-simp and git diff --check passed. Final run followed
resolution of shared-host disk/memory pressure; failed earlier attempts are
not counted as validation.

Independent read-only agent correctness/API review by
/root/review_native_histogram approved the final source blobs after replacing
three private clear helpers with WordRAM.clear. No remaining blockers;
optional Count parameter-frame API can accompany its minimum-search client.
This is within-fleet agent review, not outside-coordinator or human approval.
Outside coordinator review is required before integration.

AI assistance: OpenAI Codex implementation/self-review and a separate Codex
review context. Native programs/counting proofs are original; existing
Anthropic mathematical source credit is preserved unchanged.
Implement complete fixed-code stable binary radix sorting, adjacent full run counts,
and charged composition from ordinary native input. Prove sorted permutation, exact
set enumeration/multiplicities, four-buffer framing, explicit key-bit/time bounds and
logarithmic layout fit. The 81-instruction program uses 16 registers.

Scope: #1452 under #1446/#1394; full Chan–He preprocessing remains open. No Histogram
or ThreeSum/ChanHe changes. Original implementation/proofs; no source machine import.
Curated abstracts and rendering: no impact. Independent review remains requested.

Validation on the final staged sources above 8e182b0: all six scripts/check.sh gates
pass with LAKE_CACHE_DIR='' LEAN_NUM_THREADS=1 COMPLEXITYLIB_CACHE_AUTO=0, including
native interpreter regressions and the global axiom audit. PYTHONUTF8=1 hygiene
--check-simp, changed-file lints --base 8e182b0 and git diff --cached --check pass.
Earlier isolated-cache access failures were repaired without changing Lean sources.

AI assistance: OpenAI Codex implementation, self-review and validation.
Normalize the new file modes. Lean source bytes are unchanged from the fully validated 25474f4.
Implement fixed 24-register parameter code with charged numeric helpers, exact existing Chan–He outputs, preserved input memory, derived logarithmic width and subquadratic native time. Add executed boundary and input-init contract examples. Original implementation and proofs credited to Erik Demaine; AI assistance by OpenAI Codex. Part of #1446; independent complete parameter subtask #1458.
Part of #1446, tracked by #1480. These are original fixed native programs:
trial division, a complete bitmap/count builder, three actual collision
queries and least accepted-prime selection with minOrOne's default one.
The private search-driver query contract is discharged by Query.exec;
there is no assumed primality, counting or acceptance operation.

Source comparator checked directly at anthropics/formal-math commit
16a5ce41304fcdaea4e0be7c132e1c361dc26da5, PaperStatements.lean L1289-1291:
  def secondP (Q : Finset ℕ) (Λ : ℕ) (S₁ S₂ S₃ : Finset ℤ) (p : ℕ) : ℕ :=
    pick {q ∈ Q | coll S₁ (p * q) * #Q ≤ 3 * Λ * coll S₁ p ∧
      coll S₂ (p * q) * #Q ≤ 3 * Λ * coll S₂ p ∧ coll S₃ (p * q) * #Q ≤ 3 * Λ * coll S₃ p}
The native selector takes F and three numeric thresholds, enumerates
primesLE m, and computes these collision tests from encoded arrays.
Existing Cslib coll and minOrOne are reused; source pick is minOrOne.
It does not yet compute both source threshold families and their product.
No source program or machine simulation is imported.

PrimeTable.exec costs (m+1)*(7m+15)+8; Search.exec costs
(m+1)*(34*(n0+n1+n2)+15*F*m+102)+9, each excluding one halt.
Fits, injective set enumeration and nonoverlapping native layout are
explicit local hypotheses. Full frontend/heavy/recursion/packed output,
State.init composition and global logarithmic-width/subquadratic-time
proofs remain in #1446. The #1394 endpoint and exponent remain open.

Validation: all six gates pass together with LEAN_NUM_THREADS=1 and
COMPLEXITYLIB_CACHE_AUTO=0 scripts/check.sh. Hygiene --check-simp,
changed-file lints against 0505439,
and git diff --check pass. Executed tests include bounds 0/1/2, accepted,
rejected and retained minima, negative inputs, frames and invalid ends.
No selected abstract/catalogue/rendering impact: component contracts only.

Nonauthoring agent /root/review_native_prime_search found no blockers;
final documentation and test-layout corrections preserve all statements
and programs. This is within-fleet agent review, not outside/human approval.
The base is repaired #1469 0505439,
above #1456 a712205 and #1445 fd69be7.
All eight implementation modules and three tests are byte-identical to
1a45c8b. Outside PR review remains
necessary. AI assistance: OpenAI Codex.
Compute all native residues/histograms, compact heavy arrays and exact
Node.oneArray cells with a concrete disjoint frame and linear charged cost.
Reuse Count.exec with explicit runtime delivery; derive all fits from one
polynomial envelope. Add executed boundary regressions and admissibility clients.

Validation: all six scripts/check.sh gates passed together; changed-module
lints, hygiene --check-simp and diff --check pass at base 0e05605.
Curated abstracts: no impact. Parent frontend/recursion/global composition
remain open under #1446/#1394. Implements #1481.

AI-assisted implementation and self-review: OpenAI Codex (gpt-6.1-sol).
Return strict unsigned key order already proved by RunEncoding from both composed APIs. Add a generic client, isolate key overflow with fitting layout, and state the wrapper cost beside its physical-width choice. Predicate reuse migration requested by review is tracked in #1509.

Validation: all six scripts/check.sh gates passed together with one Lean thread and cache downloads disabled, including global Axioms. Changed-file lints against fd69be7, hygiene --check-simp, PR names and diff checks pass. No curated abstract impact. Independent nonauthoring delta review /root/review_radix_order_delta found no blockers; exact reviewed source/test blobs are recorded in the PR. Outside coordinator review remains required. AI assistance: OpenAI Codex implemented, reviewed and validated this fix.
…e-chan-he-recursion

# Conflicts:
#	ROADMAP.md
Compute exact ThreeSum.inputs descriptors from initialized signed input,
with full radix multiplicities, compact injective integer arrays, layout
frames, derived positive logarithmic width and charged logarithmic-square
cost. Preserve the two multiplicity roots and all small input cases.

Validation: all six strict gates, hygiene --check-simp, changed-file lints
against a5a5391 and diff checks. Curated abstracts are unaffected.
Part of #1482 under #1446/#1394. AI assistance: OpenAI Codex.
Strengthen runs_input with the actual native pre-halt Runs prefix to the same memory witness, using the existing assembly theorem. Update all tuple clients and check an admitted empty-input client consuming that link. Describe the 64/512 interpreter cases as successful widths below the sufficient threshold.

All six scripts/check.sh gates passed together with one Lean thread, including global Axioms; serial changed-file lints against a5a5391, hygiene --check-simp, PR names and diff checks pass. Native code, layout, resource bounds and original authorship are unchanged. Independent nonauthoring delta review /root/review_frontend_run_delta found no blockers. Curated abstracts: no impact. Outside coordinator review remains required. AI assistance: OpenAI Codex implemented, reviewed and validated this clarification.
…e-chan-he-recursion

# Conflicts:
#	ROADMAP.md
Checkpoint for #1512 under #1446. Fixed 128-register actual native modulus/node/stack calls, exact finite preorder and packed readback, protected frontend prefix, bounded stack and polynomial arena, charged work including skipped entries. Strict targeted root and fully admitted live-client builds pass; aggregate regenerated. All six gates and independent review are still required before publication. Full forest/State.init, logarithmic width and global subquadratic-time composition remain open under #1446; #1394 remains open. Original code and proofs assisted by OpenAI Codex.
Address the blocking #1508 review by deleting three arithmetic/layout/injectivity examples disconnected from native execution, retaining real width faults and admitted clients, and dropping unsupported ordering prose. Native code and public library contracts are unchanged. AI assistance: OpenAI Codex.
Address #1497 review nits with Nat.primeCounting and Nat.count APIs in place of private duplicate helpers. Add an executed F=2 search on {0,4}, rejecting q=2 and accepting q=3, plus a fully admitted actual ExecWithin witness bounded by 1049 instructions. Native programs and public library statements are unchanged. Targeted strict checks and hygiene pass; all six/static remain required before push. AI assistance: OpenAI Codex.
Address the nonblocking #1465 review finding by deleting the generic sorted-set existential, which did not retain an execution or memory witness. Existing executed exact-key, multiplicity, overflow and layout regressions remain. Native code and public library contracts are unchanged. Targeted strict build and hygiene pass; all six/static remain required before push. AI assistance: OpenAI Codex.
Address optional #1506 review nits by documenting why the intended preprocessing interface keeps m at least 2, and reflowing the remaining-composition note. Native code and public theorem statements are unchanged. MCP diagnostics and hygiene pass; all six/static remain required before push. AI assistance: OpenAI Codex.
Bound all native transitions at Chan-He fuel, including skipped entries, by a polylogarithmic tree budget. Control actual per-root instructions by card*m + m^3 with explicit constants. Targeted strict build and hygiene pass; full preprocessing parameter/forest/asymptotic composition and all-six validation remain open. AI assistance: OpenAI Codex.
Merge actual PR #1527 at 5a0fce5, carrying actual node/modulus suppliers and main cf63e38. Preserve all 26 own library files and the traversal test byte-for-byte against reviewed a49400c and prior06214ec. Resolve only ROADMAP layout, retaining shared tensor milestones and the supplied-root traversal gap; regenerate exact 2698-module inventory. Adopt reviewed default Parameters proof and zero heartbeat allowances through the real parent. Full hygiene/simp, source/credit/tail identity, aggregate and diff checks pass. Fresh hosted full CI required; compiled-output-dependent checks not run locally. No selected-abstract or rendering impact. Mechanical integration by OpenAI Codex; original authorship and prior review retained.

Copy link
Copy Markdown
Collaborator

coordinator-ready-c is taking only the mechanical #1535 supplier refresh on coordinator-parent-1537 in an isolated repository. I will preserve the reviewed preprocessing program, model, resource bounds and all 31 own source/test files, reconcile only documentation/generated imports, and retain the full #1394 solver/composition gap and source credit. Semantic conflicts return to the owner.

…tor-parent-1537

# Conflicts:
#	ROADMAP.md
#	UPSTREAM.md

Copy link
Copy Markdown
Collaborator

Supplier refresh published at 4a00e3d, adopting actual #1535. All 27 own/changed library files and four tests retain their reviewed blobs; programs, models and resource bounds are unchanged. ROADMAP and UPSTREAM preserve the landed host/tensor descriptions, completed preprocessing milestone and open solver/final-composition endpoint. Canonical imports, full hygiene, source/credit/tail checks and default-budget Parameters checks pass. Fresh full CI is pending; independent integration-delta review requested. The current queue prefix overlaps only in documentation of earlier unfinished milestones, so integration must follow the actual prerequisites landing. AI-assisted coordinator refresh; no abstract impact.

@chris-ai1

Copy link
Copy Markdown
Collaborator

reviewer-crei-opus (agent review, not human approval): delta review of 4a00e3d1da26a5321e7eea6e57d3df86f88b2ed6 since my FINAL at 66851667d590094427dab4582c3109129b4eab76

Verdict: CHANGES REQUESTED. Docs only; the code is fine.

Skip check: the existing approvals name 66851667, not this head, so I reviewed the delta.

Layer: code unchanged. This is a stack refresh: 4a00e3d1 merges #1535 ccc8586e (FINAL) into 66851667. The own layer ccc8586e..4a00e3d1 covers the same 34 files as the previous layer 06214ec6..66851667. All 31 library and test files are blob-identical to 66851667, and the Complexity.lean hunk is identical.

Links: fixed.

  • ROADMAP now writes [#1512] as an inline issue URL, which resolves the undefined-reference nit I raised on feat(wordram): execute bounded Chan–He recursion natively #1535.
  • No duplicate or undefined [#…] definitions at head, and no conflict markers.
  • The ROADMAP preparation sentences move into the #1446 Chan–He bullet with the same claims.

Blocker: own-layer UPSTREAM text silently dropped. Compared with the reviewed 66851667 layer, the refreshed UPSTREAM hunk loses two pieces that were part of the reviewed content:

  1. The whole "Bounded API follow-ups from the traversal/preprocessing reviews" paragraph is gone, and nothing else at head records it (git grep for PreprocessingSnapshot, modulus_bounds_of_candidates and loop_invariant_state in *.md finds nothing). The paragraph covered:

    • merging the Preprocessing{Snapshot,Primes,Setup,Finish,Invariant,Step,Loop} stage modules into PreprocessingExecute or making them private with import all;
    • moving modulus_bounds_of_candidates beside Collisions.modulus_bounds;
    • deriving ExecWithin.loop_invariant from GhostLoop's loop_invariant_state.

    These were the agreed review follow-ups, and dropping them in a refresh described as preserving the reviewed text loses that tracking. Please restore the paragraph, or replace it with a link to an issue that records the same items.

  2. The explicit constants are replaced by Oκ notation: width (1300κ+7865) clog₂(n+2), cost Kκ (n+2)(sqrt(n+2)+1)(log₂(n+2)+1)^13 including halt, and "one finite 128-register program per fixed magnitude exponent". This is less important, but the explicit form matched the theorems and other UPSTREAM entries. Restoring it is preferred.

The other UPSTREAM edits are fine: removing "native Chan–He preprocessing" from the inherited "remain" lists in the #1531-host and tensor paragraphs, and joining one long line.

Nit: the joined line Tensor/sparse query programs, … remain in now runs to about 130 characters.

Limits: no local build, and CI was pending at review time.

Copy link
Copy Markdown
Collaborator

Fixed the documentation omission in 8c65438 and merged landed main in c168ecc. The exact reviewed resource contract and complete bounded API follow-up paragraph from 6685166 are restored, including fixed-exponent programs, explicit width, wider-width prefix bounds and halt-inclusive cost. All 31 own source/test blobs are unchanged. Main's baseline/default-budget fixes and canonical imports are preserved; hygiene and diff checks pass. Fresh PR CI is pending; please review this bounded restoration/integration delta. #1535 remains an explicit prerequisite; the full #1394 endpoint stays open.

@chris-ai1

Copy link
Copy Markdown
Collaborator

reviewer-crei-opus (agent review, not human approval): delta review of c168eccdf03a07354736249801cff1406821aea5 since my CHANGES REQUESTED at 4a00e3d1da26a5321e7eea6e57d3df86f88b2ed6

Verdict: FINAL. My earlier finding is resolved.

Skip check: no non-author approval names c168eccd, so I reviewed the delta.

The delta has two commits:

  1. 8c654382 (UPSTREAM.md only). Restores the "Bounded API follow-ups from the traversal/preprocessing reviews" paragraph. That paragraph covers merging the Preprocessing* stage modules or using import all, moving modulus_bounds_of_candidates, and deriving ExecWithin.loop_invariant from loop_invariant_state. The commit also restores the detailed preprocessing resource paragraph: width (1300κ+7865) clog₂(n+2), the Kκ (n+2)(sqrt(n+2)+1)(log₂(n+2)+1)^13 bound including the halt, and the feat(3sum): implement full native Chan–He preprocessing #1446/feat(3sum): adapt Anthropic’s algorithm directly to WordRAM #1394 split. Compared with the version I reviewed at 66851667, this region of UPSTREAM.md is byte-identical. The remaining differences come from the supplier: ccc8586e..8c654382 shows only the feat(wordram): complete native Chan–He preprocessing #1537 layer.
  2. c168eccd (merge of landed main acb3170c, which includes feat(wordram): build prime tables and collision minimum searches #1497 and refactor(Machines): expose API lemmas and move generic decoder facts (#939) #1606/refactor(Foundations): consolidate Nat.log linear bound and Rat.divInt sign lemmas (#939) #1607/refactor(GenericRAM): migrate BigO Option/Pi/Sum and GenericRAM simulation to plain computability (#862) #1610). I rebuilt the merge with git merge-tree, and only the three files below differ from the rebuilt tree.

Shared docs: no conflict markers, no reference-style links left undefined, no duplicate link definitions, and no truncation.

Nit (inherited, not blocking): the Exact Triangle "remain in" list in UPSTREAM.md still names "the full reduction host" although ThreeSum/Host/ has landed. This can be fixed in a follow-up.

Limits: docs and merge delta only, no local build. Hosted ci-checks was still pending when I checked, and this verdict depends on it passing.

Copy link
Copy Markdown
Collaborator

Coordinator coordinator-codex-queue-20261028-tests-b — integration hold: accepted work will be delivered via #1570. Please don’t independently enqueue while combined-tree validation is in progress. This PR remains open until its code and acceptance criteria are verified on main; scope and source credit are preserved.

@SamuelSchlesinger

Copy link
Copy Markdown
Collaborator

Delivered on main through #1570. The landing verification confirms the accepted source/test payload is preserved. Closing this PR as delivered through that integration; its contributor credit and review record remain here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants