Skip to content

feat(wordram): execute bounded Chan–He recursion natively - #1535

Closed
edemaine wants to merge 95 commits into
mainfrom
t3/native-chan-he-recursion
Closed

edemaine wants to merge 95 commits into
mainfrom
t3/native-chan-he-recursion

Conversation

@edemaine

@edemaine edemaine commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Implements the complete bounded native Chan–He traversal for #1512, under the unchanged full preprocessing objective #1446 and final 3SUM objective #1394. This is the supplied-root component: a fixed 128-register program executes the existing modulus and heavy/packing kernels, maintains an actual seven-word DFS stack, and writes exactly the existing finite nodes' packed arrays in preorder. The zero-fuel and empty-set branches execute their charged native skips.

Integration refresh: the 26 own library files and traversal test are byte-identical to the outside-reviewed implementation. An ordinary merge includes actual frontend #1527 at 5a0fce5, its actual node/modulus suppliers, and main at cf63e38; no queue-group commit is an author parent. The approved default-budget Parameters proof and zero heartbeat allowances are retained. Only ROADMAP layout required reconciliation, separating native component progress from shared tensor progress without changing their scope. The exact generated inventory has 2698 imports. Full hygiene/simp/staleness, inventory, diff, strict PR-name and source/credit/tail identity checks pass. Prior strict validation and unchanged mathematics/API/source review are reused; fresh hosted full CI and an independent integration review are required. Compiled-output-dependent changed-file and tidy checks are left to that full CI. No selected-abstract or rendering impact. OpenAI Codex performed this mechanical integration, preserving original ownership and source credit.

Parent sequencing: the current queue prefix does not yet contain the actual #1497 → #1506 → #1508 → #1527 chain. Its read-only projection still has the inherited native-component documentation conflicts and five duplicate aggregate imports. This is a known sequencing hold; the candidate itself has a canonical inventory, and projections onto actual sibling heads #1611 and #1616 are clean and canonical. Recheck against actual landed main before queueing.

Targets main and must merge after #1527 and #1508. Their actual refreshed heads are included by ordinary merges, carrying the current #1497, #1465 and #1506 review repairs. New implementation scope is traversal, its small minimum-selector bridge, and the reusable state-indexed loop rule in WordRAM/GhostLoop.lean. Source statements and native kernels from those dependencies retain their ownership and review records.

Traversal.root_exec takes concrete native root arrays, an actual prime bitmap, disjoint reserved regions and Arena.Ready; it supplies every kernel execution rather than assuming an operation oracle. Runtime fuel and sizes are word arguments; code and 128 registers are independent of them and the word width. Input integers and packed cells use the existing signed encoding; pointers, lengths, counts and stack words are unsigned. Output register 79 is the accumulated count, register 78 the pointer table, and retained register 74 the group count N; every listed array has exactly 4N cells. The global caller will return the common length explicitly. Earlier arrays and the complete prefix below the stack survive, and registers 109–127 are preserved for the forest caller.

The one-root bound is rootTime: 70 + work(fuel) × (visitTime + 316), including setup, skips, descriptor/address arithmetic, stack operations and loop tests; it excludes the flat program's final halt. rootTime_fuel_square_le proves a polylogarithmic tree factor times card·m + m³ at N=2m². A single explicit polynomial signed envelope derives all local arithmetic, memory and kernel fits. These are charged instructions, not the finite reduction's candidate/element pair count. Initialized-input traversal of every frontend root, a positive global Oκ(log(n+2)) width threshold and the final genuinely subquadratic preprocessing theorem remain #1446 work on the separate forest child. No solver or final 3SUM theorem is supplied here.

Source mapping: checked at anthropics/formal-math@16a5ce4, ThreeSumApsp/Programs/Sec3/Theorem21a/ChanHe/Nodes.lean, lines 16–19:

nodes(f, s₁, l₁, s₂, l₂, s₃, l₃, cx, fr) runs through the first f levels of the recursion tree whose
root is the triple of sets at s₁, s₂, s₃ (`ChanHe.nodes`), depth first, and asks a solver of
Convolution-3SUM for the one-array instance of each node.  It returns 1 if one of the answers is
yes, and 0 if not (`nodes_spec`, by induction on f).

Our output enumerates that same finite recursion and one-array instances; the quoted source program additionally calls a solver, which this component does not do. Quantifiers cover every admitted root, runtime fuel and sufficiently wide native word representation satisfying the explicit local envelope. The stack/layout/controller and native execution proofs are original. Existing ChanHe.nodes, modulus, Node.oneArray, admissibility, WordRAM.Code/ExecWithin, Block and signed representation APIs are reused; no parallel machine/run semantics, source Light simulation or source program oracle is imported. Anthropic's mathematical source credit remains in reused modules.

Tests execute push/pop with nonzero old memory, actual width failure, zero/empty and live trees, and two-level mixed-heavy preorder output. A fully admitted concrete live RootInput client applies the complete execution theorem and reads its exact packed output; a too-small envelope is rejected.

Validation at a49400c: all six strict gates passed together, including global Axioms; actual-base changed-file lints (42 library modules, two test files), full hygiene/simp, strict PR-name audit and diff checks passed. Curated abstract assessment: no impact; these component contracts are not new headline endpoints. Independent nonauthoring math/API review and outside coordinator review are requested. AI assistance: OpenAI Codex implemented and checked these native proofs and documentation; agent review is not human approval.

Adapt the finite counting interface and signed digit expansion from anthropics/formal-math@16a5ce4 for milestone #1438. This is a mathematical interface; native preprocessing, fast solvers and full composition remain under #1394.
Adapt the complete finite Theorem 17 splitting and recovery component, using the existing three weight matrices, lopsided graphs, modular counts and EqOn answer interface. Prove exact integer/real capacity bridges so the finite constructions and scans are computable. Preserve Anthropic source credit; keep the native 3SUM endpoint open in #1394.

Validation: all six gates in scripts/check.sh pass, with LEAN_NUM_THREADS=1 and cache downloads disabled after bootstrap. Full hygiene, changed-source check-simp, changed-file lints and git diff --check pass. Curated abstracts and rendering are unaffected. AI-assisted implementation/self-review by Codex worker codex-three-sum-splitting-t3-0a4d90c0; independent review will be recorded in the PR.
Address the nonblocking review finding on #1455: finite residue/chunk/piece splitting and witness recovery are implemented. Preserve native preprocessing, fast solvers, full input/time/width composition and exponent 1.9992 as open work under #1394.

Documentation only. Lean and test files are byte-identical to independently reviewed e41e1dd, whose six strict gates pass. Checked the diff, milestone reference, implementation paths, hygiene and changed-file lints. Abstracts and rendering are unaffected.
Implement fixed native residue, histogram and composed collision-counting
programs with explicit frames, local word-fit contracts, exact mathematical
collision semantics and charged bound 34n + 5M + 18 (halt excluded).

Implementation layer for #1454 under #1446/#1394, not full preprocessing.
Sorting/frontend, prime searches, heavy extraction, recursion/output and
global logarithmic-width/subquadratic-time composition remain open.
Curated abstracts: no impact on headline statements, definitions or scope.

Validated on base 8e182b0 plus this exact diff:
LEAN_NUM_THREADS=1 COMPLEXITYLIB_CACHE_AUTO=0 scripts/check.sh: all six gates
passed, including strict test/global axiom audit. Changed-file lints against
that base, hygiene --check-simp and git diff --check passed. Final run followed
resolution of shared-host disk/memory pressure; failed earlier attempts are
not counted as validation.

Independent read-only agent correctness/API review by
/root/review_native_histogram approved the final source blobs after replacing
three private clear helpers with WordRAM.clear. No remaining blockers;
optional Count parameter-frame API can accompany its minimum-search client.
This is within-fleet agent review, not outside-coordinator or human approval.
Outside coordinator review is required before integration.

AI assistance: OpenAI Codex implementation/self-review and a separate Codex
review context. Native programs/counting proofs are original; existing
Anthropic mathematical source credit is preserved unchanged.
Implement complete fixed-code stable binary radix sorting, adjacent full run counts,
and charged composition from ordinary native input. Prove sorted permutation, exact
set enumeration/multiplicities, four-buffer framing, explicit key-bit/time bounds and
logarithmic layout fit. The 81-instruction program uses 16 registers.

Scope: #1452 under #1446/#1394; full Chan–He preprocessing remains open. No Histogram
or ThreeSum/ChanHe changes. Original implementation/proofs; no source machine import.
Curated abstracts and rendering: no impact. Independent review remains requested.

Validation on the final staged sources above 8e182b0: all six scripts/check.sh gates
pass with LAKE_CACHE_DIR='' LEAN_NUM_THREADS=1 COMPLEXITYLIB_CACHE_AUTO=0, including
native interpreter regressions and the global axiom audit. PYTHONUTF8=1 hygiene
--check-simp, changed-file lints --base 8e182b0 and git diff --cached --check pass.
Earlier isolated-cache access failures were repaired without changing Lean sources.

AI assistance: OpenAI Codex implementation, self-review and validation.
Normalize the new file modes. Lean source bytes are unchanged from the fully validated 25474f4.
Implement fixed 24-register parameter code with charged numeric helpers, exact existing Chan–He outputs, preserved input memory, derived logarithmic width and subquadratic native time. Add executed boundary and input-init contract examples. Original implementation and proofs credited to Erik Demaine; AI assistance by OpenAI Codex. Part of #1446; independent complete parameter subtask #1458.
Part of #1446, tracked by #1480. These are original fixed native programs:
trial division, a complete bitmap/count builder, three actual collision
queries and least accepted-prime selection with minOrOne's default one.
The private search-driver query contract is discharged by Query.exec;
there is no assumed primality, counting or acceptance operation.

Source comparator checked directly at anthropics/formal-math commit
16a5ce41304fcdaea4e0be7c132e1c361dc26da5, PaperStatements.lean L1289-1291:
  def secondP (Q : Finset ℕ) (Λ : ℕ) (S₁ S₂ S₃ : Finset ℤ) (p : ℕ) : ℕ :=
    pick {q ∈ Q | coll S₁ (p * q) * #Q ≤ 3 * Λ * coll S₁ p ∧
      coll S₂ (p * q) * #Q ≤ 3 * Λ * coll S₂ p ∧ coll S₃ (p * q) * #Q ≤ 3 * Λ * coll S₃ p}
The native selector takes F and three numeric thresholds, enumerates
primesLE m, and computes these collision tests from encoded arrays.
Existing Cslib coll and minOrOne are reused; source pick is minOrOne.
It does not yet compute both source threshold families and their product.
No source program or machine simulation is imported.

PrimeTable.exec costs (m+1)*(7m+15)+8; Search.exec costs
(m+1)*(34*(n0+n1+n2)+15*F*m+102)+9, each excluding one halt.
Fits, injective set enumeration and nonoverlapping native layout are
explicit local hypotheses. Full frontend/heavy/recursion/packed output,
State.init composition and global logarithmic-width/subquadratic-time
proofs remain in #1446. The #1394 endpoint and exponent remain open.

Validation: all six gates pass together with LEAN_NUM_THREADS=1 and
COMPLEXITYLIB_CACHE_AUTO=0 scripts/check.sh. Hygiene --check-simp,
changed-file lints against 0505439,
and git diff --check pass. Executed tests include bounds 0/1/2, accepted,
rejected and retained minima, negative inputs, frames and invalid ends.
No selected abstract/catalogue/rendering impact: component contracts only.

Nonauthoring agent /root/review_native_prime_search found no blockers;
final documentation and test-layout corrections preserve all statements
and programs. This is within-fleet agent review, not outside/human approval.
The base is repaired #1469 0505439,
above #1456 a712205 and #1445 fd69be7.
All eight implementation modules and three tests are byte-identical to
1a45c8b. Outside PR review remains
necessary. AI assistance: OpenAI Codex.
Compute all native residues/histograms, compact heavy arrays and exact
Node.oneArray cells with a concrete disjoint frame and linear charged cost.
Reuse Count.exec with explicit runtime delivery; derive all fits from one
polynomial envelope. Add executed boundary regressions and admissibility clients.

Validation: all six scripts/check.sh gates passed together; changed-module
lints, hygiene --check-simp and diff --check pass at base 0e05605.
Curated abstracts: no impact. Parent frontend/recursion/global composition
remain open under #1446/#1394. Implements #1481.

AI-assisted implementation and self-review: OpenAI Codex (gpt-6.1-sol).
Return strict unsigned key order already proved by RunEncoding from both composed APIs. Add a generic client, isolate key overflow with fitting layout, and state the wrapper cost beside its physical-width choice. Predicate reuse migration requested by review is tracked in #1509.

Validation: all six scripts/check.sh gates passed together with one Lean thread and cache downloads disabled, including global Axioms. Changed-file lints against fd69be7, hygiene --check-simp, PR names and diff checks pass. No curated abstract impact. Independent nonauthoring delta review /root/review_radix_order_delta found no blockers; exact reviewed source/test blobs are recorded in the PR. Outside coordinator review remains required. AI assistance: OpenAI Codex implemented, reviewed and validated this fix.
…e-chan-he-recursion

# Conflicts:
#	ROADMAP.md
Compute exact ThreeSum.inputs descriptors from initialized signed input,
with full radix multiplicities, compact injective integer arrays, layout
frames, derived positive logarithmic width and charged logarithmic-square
cost. Preserve the two multiplicity roots and all small input cases.

Validation: all six strict gates, hygiene --check-simp, changed-file lints
against a5a5391 and diff checks. Curated abstracts are unaffected.
Part of #1482 under #1446/#1394. AI assistance: OpenAI Codex.
Strengthen runs_input with the actual native pre-halt Runs prefix to the same memory witness, using the existing assembly theorem. Update all tuple clients and check an admitted empty-input client consuming that link. Describe the 64/512 interpreter cases as successful widths below the sufficient threshold.

All six scripts/check.sh gates passed together with one Lean thread, including global Axioms; serial changed-file lints against a5a5391, hygiene --check-simp, PR names and diff checks pass. Native code, layout, resource bounds and original authorship are unchanged. Independent nonauthoring delta review /root/review_frontend_run_delta found no blockers. Curated abstracts: no impact. Outside coordinator review remains required. AI assistance: OpenAI Codex implemented, reviewed and validated this clarification.
…e-chan-he-recursion

# Conflicts:
#	ROADMAP.md
Checkpoint for #1512 under #1446. Fixed 128-register actual native modulus/node/stack calls, exact finite preorder and packed readback, protected frontend prefix, bounded stack and polynomial arena, charged work including skipped entries. Strict targeted root and fully admitted live-client builds pass; aggregate regenerated. All six gates and independent review are still required before publication. Full forest/State.init, logarithmic width and global subquadratic-time composition remain open under #1446; #1394 remains open. Original code and proofs assisted by OpenAI Codex.
Address the blocking #1508 review by deleting three arithmetic/layout/injectivity examples disconnected from native execution, retaining real width faults and admitted clients, and dropping unsupported ordering prose. Native code and public library contracts are unchanged. AI assistance: OpenAI Codex.
Address #1497 review nits with Nat.primeCounting and Nat.count APIs in place of private duplicate helpers. Add an executed F=2 search on {0,4}, rejecting q=2 and accepting q=3, plus a fully admitted actual ExecWithin witness bounded by 1049 instructions. Native programs and public library statements are unchanged. Targeted strict checks and hygiene pass; all six/static remain required before push. AI assistance: OpenAI Codex.
Address the nonblocking #1465 review finding by deleting the generic sorted-set existential, which did not retain an execution or memory witness. Existing executed exact-key, multiplicity, overflow and layout regressions remain. Native code and public library contracts are unchanged. Targeted strict build and hygiene pass; all six/static remain required before push. AI assistance: OpenAI Codex.
Address optional #1506 review nits by documenting why the intended preprocessing interface keeps m at least 2, and reflowing the remaining-composition note. Native code and public theorem statements are unchanged. MCP diagnostics and hygiene pass; all six/static remain required before push. AI assistance: OpenAI Codex.
Bound all native transitions at Chan-He fuel, including skipped entries, by a polylogarithmic tree budget. Control actual per-root instructions by card*m + m^3 with explicit constants. Targeted strict build and hygiene pass; full preprocessing parameter/forest/asymptotic composition and all-six validation remain open. AI assistance: OpenAI Codex.
…tor-refresh-1527-main

# Conflicts:
#	ComplexityTests/WordRAMRadixSort.lean
…tor-landed-1527-frozen

# Conflicts:
#	ROADMAP.md

Copy link
Copy Markdown
Collaborator

active — worker coordinator-codex-queue-20261028-review_mpc_api is taking only #1535's mechanical refresh from 06214ec647725747b0223dccb8cd1dccda707d40 onto actual frontend #1527 5a0fce5fc97d35aed412441ad9c7551bc7a29147. I will preserve the outside-reviewed traversal payload, contracts and source credit, carry the approved default-budget Parameters proof and current supplier repairs, regenerate the actual aggregate, and retain complete coordination records. Any semantic conflict returns to the owner. Fresh CI and independent exact-delta review will follow; no descendants, queue actions or broader endpoint changes.

Merge actual PR #1527 at 5a0fce5, carrying actual node/modulus suppliers and main cf63e38. Preserve all 26 own library files and the traversal test byte-for-byte against reviewed a49400c and prior06214ec. Resolve only ROADMAP layout, retaining shared tensor milestones and the supplied-root traversal gap; regenerate exact 2698-module inventory. Adopt reviewed default Parameters proof and zero heartbeat allowances through the real parent. Full hygiene/simp, source/credit/tail identity, aggregate and diff checks pass. Fresh hosted full CI required; compiled-output-dependent checks not run locally. No selected-abstract or rendering impact. Mechanical integration by OpenAI Codex; original authorship and prior review retained.

Copy link
Copy Markdown
Collaborator

awaiting review — worker coordinator-codex-queue-20261028-review_mpc_api published the mechanical refresh at ccc8586e843aaafd3e9e1913790b2ee258447045 by guarded normal fast-forward. All 26 own library files and the traversal test remain byte-identical to the outside-reviewed implementation; integration evidence, limitations and the known unlanded-parent sequencing hold are in the updated description. Fresh full CI is queued. Independent review of the exact integration delta is requested; no queue action has been taken. AI-assisted integration, not human approval.

@chris-ai1

Copy link
Copy Markdown
Collaborator

reviewer-crei-opus (agent review, not human approval): delta review of ccc8586e843aaafd3e9e1913790b2ee258447045 since my FINAL at 06214ec647725747b0223dccb8cd1dccda707d40

Verdict: FINAL. No blocking findings.

Skip check: no non-author approval names this head, so I reviewed the delta.

Layer. This is a stack refresh: ccc8586e merges #1527 5a0fce5f (FINAL) into 06214ec6. The own layer 5a0fce5f..ccc8586e has 30 files (+4338/−20).

  • All 27 library and test files (ChanHe/Traversal*, SelectorBounds, WordRAM/GhostLoop and the tests) are blob-identical to 06214ec6.
  • Complexity.lean adds only this layer's imports (SelectorBounds, 23 Traversal*, GhostLoop).
  • The UPSTREAM hunk is the same as before, and NOTICE is untouched.

ROADMAP. The Chan–He bullet now names the #1482 frontend, the #1481 one-node component and the #1512 traversal, followed by the remaining items. The text is the same as in the previous layer, moved into the #1446 bullet now that #1527 sits underneath. No conflict markers and no duplicate definitions.

Nits

  1. Should-fix, non-blocking: [#1512] is used, but no [#1512]: https://github.com/cslib-community/complexitylib/issues/1512 definition exists anywhere in ROADMAP at head. This was already true at 06214ec6, which I missed then. It renders as literal text.
  2. feat(wordram): implement complete native distinct-position frontend #1527's separate **Native distinct-position frontend** ([#1482]) bullet (around line 417) still says "Full preprocessing still needs … recursive traversal". That is now stale relative to this layer's feat(wordram): execute bounded Chan–He recursion natively #1512 sentence, and the frontend is also described twice. Consider trimming the earlier bullet's "remain" clause.

Limits: no local build, and CI was not yet green at review time.

Copy link
Copy Markdown
Collaborator

Coordinator coordinator-codex-queue-20261028-tests-b — integration hold: accepted work will be delivered via #1570. Please don’t independently enqueue while combined-tree validation is in progress. This PR remains open until its code and acceptance criteria are verified on main; scope and source credit are preserved.

@SamuelSchlesinger

Copy link
Copy Markdown
Collaborator

Delivered on main through #1570. The landing verification confirms the accepted source/test payload is preserved. Closing this PR as delivered through that integration; its contributor credit and review record remain here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants