Repository navigation
feat(Probability): pair and tail estimates for frame-conditioned retention channels - #1710
SamuelSchlesinger merged 40 commits into
Conversation
Advance CSLib to 8a8ad748 and synchronize Mathlib, the documentation generator, Verso, and all three lockfiles with Lean v4.36.0-rc2. Keep the root and documentation projects on the same inherited dependency revisions.
Replace 187 deprecated ENNReal wrapper uses across 55 proof files with their generic counterparts. The exact mapping is: - ENNReal.tsum_le_tsum -> tsum_le_tsum (94) - ENNReal.tsum_add -> tsum_add (54) - ENNReal.tsum_eq_zero -> tsum_eq_zero_iff (14) - ENNReal.summable -> CompleteLattice.summable (6) - ENNReal.tsum_comm -> tsum_comm (6) - ENNReal.tsum_prod' -> tsum_prod' (5) - ENNReal.le_tsum -> le_tsum (5) - ENNReal.tsum_eq_add_tsum_ite -> tsum_eq_add_tsum_ite (2) - ENNReal.tsum_sigma -> tsum_sigma (1) The old wrappers invoke these same theorems specialized to ENNReal. All five le_tsum calls already pass the function by name. No statements or definitions change.
Qualify newly protected matrix and finite-sum lemmas, publicly import the relocated Matrix.Array API, and replace renamed angle, parity, trailing-coefficient and polynomial-degree helpers. The generic polynomial lemmas retain the original field specializations and exponent hypotheses. No theorem statement or definition body changes. Validation: strict focused builds of all ten affected modules passed under Lean v4.36.0-rc2. Source comparison against d6ccb24 shows all Lean edits are inside theorem proofs except the required Bird import.
Preserve the existing non-unitary matrix counterexample under rc2 Mathlib by explicitly selecting Matrix.mul_apply, as in the library proof repairs.
Use expected-type-directed refinement for the generic zero-sum lemma and expose the determinant variable-count function before fun_prop. Normalize graph and Cook-Levin wrappers in regression proofs while retaining direct/let-bound automation and all missing-hypothesis checks. Compose the existing rotation certificates for arithmetic-base examples, and remove one redundant concrete Fintype binder from a private test helper. Validation: strict focused builds of the two source and five test modules passed under Lean v4.36.0-rc2. No public theorem statement, definition body, model, encoding, probability, or resource bound changes. No heartbeat or fun_prop compatibility option was introduced.
…ib-rc2-sync-1648-codex
…transition matrices Pure linear algebra for the correction-and-copy hierarchy (cslib-community#1444 Phase 2, ideas 2 and 4 of cslib-community#474), following the OpenAI preprint (2026-09-23), Section 4: the factorization 1 - (C - E + EC) = (1 + E)(1 - C), the corrected inverse, copy splitting intertwined with its projection, and the reward transport identity with the stage error bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ly it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ward lower bound Review of cslib-community#1676: correction_mulVec_one_le only shows that row sums do not grow, so its docstring, the module summary and the test comment no longer call it substochasticity. Add rewardTransport_mulVec_nonneg, the lower half of (4.30), under nonnegative projections, corrections and reward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second slice of cslib-community#1444 Phase 2 (idea 3 of cslib-community#474), following the OpenAI preprint (2026-09-23), Lemma 4.1 and Lemma 4.4: the cutoff correction recursion E = phi((C + EC)) on a strictly upper triangular C has a unique fixed point, its iterates increase towards it, and after k iterations the error is at most f e^(1/theta) (L/theta)^k / k!, independently of the size of the matrix. The corrected matrix of a finite iterate is nonnegative and small in fully corrected columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review nit on cslib-community#1677: drop the private hockey-stick lemma. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of cslib-community#1677: the module docstring now bounds the chain count by m.choose k for the actual support size m, as the proof does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ommunity#1444) A point is retained when its hash lies in a target set B. Under a pairwise independent family each point is retained with probability #B/|Y|, two distinct points independently; the number of retained points of a set has mean #S p, variance #S p (1 - p), Chebyshev's inequality and the lower tail bound 4 / (#S p). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Third slice of cslib-community#1679 (Phase 2 of cslib-community#1444, ideas 3 and 4 of cslib-community#474), following the OpenAI preprint (2026-09-23), Lemma 4.2, Lemma 4.4 and Proposition 4.5: row-sum, entry, domination and support bounds for cutoff corrections; the copy step with explicit split fractions, its density and active-copy bounds; incidence counting; and the one-stage rates that feed the stage bound, given an abstract detector interface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…b-community#1682) Identifiers y : ZMod P are retained when A y + b lies in the box {v | all coordinates < L} of ZMod P ^ H. Exact rates (L/P)^H and their squares for pairs, the lower tail 4/(#A pi), independence of separate seeds, and the assembled channel properties at rate f = a^H with L = ceil(P a): f <= pi <= 2^H f. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…2 pin (0d38523) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under rc2 the root-namespace add_apply/neg_apply/sub_apply/mul_apply/one_apply no longer apply to matrices; use the Matrix.* versions in CorrectionHierarchy and its test, as cslib-community#1653 does elsewhere. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add cutoffCorrectionIter_isStrictUpperTriangular, cutoffCorrection_residual_isStrictUpperTriangular and copyStage_isStrictUpperTriangular (for the lexicographic order on Lex (V × Fin (m + 1))), so a stage's output can feed the next stage. Correct two descriptions: the pilot inputs have bounded row sums, not necessarily at most one, and copyStage_mulVec_one_le preserves the row-sum bound, with nonnegativity from copyStage_nonneg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for rc2 Under rc2 the root-namespace add_apply/sub_apply/mul_apply no longer apply to matrices; use the Matrix.* versions in FactorialTruncation and its test, rewrapping lines that crossed 100 columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de copySplit State the preservation lemma for arbitrary weights and proportions in CorrectionHierarchy.lean, for the lexicographic order on Lex (V × ι), and derive copyStage_isStrictUpperTriangular from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… conditional averaging (cslib-community#1682) Frame-conditioned retention channels over ZMod P with their exact marginal rate, coordinates (F, w) on retention slices, the affine move (F, w) -> (gF, gw + c), validated walk variants with their unique wrap, and conditional variance decay on every slice from a spectral-gap hypothesis on the coordinates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…istic column detectors Phase 2 slice 4 of cslib-community#1444 (tracked in cslib-community#1679): the pilot of Lemma 4.2 with step-dependent multipliers, the column detector of Lemma 4.3 over an abstract retention-channel interface, and one stage of the hierarchy whose multipliers, copy weights and overloaded set come from the detector, discharging the detector interface of copyStage_rates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CorrectionSeq The row-sum, entry and input bounds of cutoffCorrectionIter in CopyStage are the fixed-multiplier case of the correction-sequence bounds; reuse them instead of re-proving. Statements are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tectors-affine-1444 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erface Instantiate RetentionChannels (cslib-community#1696) with the affine channels of cslib-community#1688 at rate a^H, with c_pi = 1, C_pi = C_pair = 2^H and C_var = 4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto feat/frame-retention-1444
…orrection for rc2 Under rc2 the root-namespace add_apply/sub_apply/mul_apply no longer apply to matrices; use the Matrix.* versions, as rc2-refresh/1676 and /1677 do for the lower modules of this stack. Proof-only; no statement changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntion channels Lemma 3.1, items 2-3, of the OpenAI preprint for frame-conditioned channels: the frame density q_fr (with 1/2 <= q_fr for r < H, via Mathlib's card_linearIndependent), pairwise independence of the unrestricted polynomial channel, the transfer bound dens_frame <= dens / q_fr, the pair bound and the Chebyshev lower tail, and the detector-interface instance retentionChannels_frameRetains (constants 2^H / q_fr and 4 / q_fr, or 2^(H+1) and 8 for r < H). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edemaine
left a comment
There was a problem hiding this comment.
reviewer-codex-edemaine (independent agent review, not human approval): APPROVE at e6b85378cab8203315c44bece13f456b566685cb.
Reviewed the complete own layer, parent merges and rc2 proof repairs, reusing the existing #1698/#1691 coverage. No blocking findings. The frame count and density formula are correct; the induction gives q_fr ≥ 1/2 for r < H. Pairwise independence is used only before conditioning, with the conditioning loss 1/q_fr applied to the joint event and Chebyshev tail. The detector constants correctly incorporate rate rounding and that loss.
Checked the source quotations/mapping against pinned Lemma 3.1 (pp. 9–10). Pair/tail results require prime P and r ≥ 1; the usable detector instance has r ≤ H, and its field-independent constants use r < H. The F₃ counterexample correctly demonstrates failure of pairwise independence after conditioning (joint probability 1/3); density/count tests and the F₅ detector client instantiate the interface. Affine-substitution invariance remains outside this PR. API placement/reuse is appropriate; no curated-abstract impact.
Lean CI was running when inspected; title checks passed. The author reports all six local gates passing; I ran no local Lean build. Live head rechecked before posting.
Preserve the complete eight-path cslib-community#1696 feature/refactor over actual main 311e984. Resolve four inherited parent files with validated main and retain the original 80b604b CopyStage refactor. All seven nonaggregate own blobs match the reviewed source before the separate credited qualification repair. All unrelated main paths and the canonical import union are retained. The own-layer whitespace check against actual main passes. A first-parent check reports only the unchanged, inherited CuckooHashing EOF blank line; that unrelated main blob is preserved rather than cleaned up here. AI-assisted coordinator integration: codex-coord-20261029-1732. Preflight credit: codex-detectors-1696-refresh-20261030. Validation pending on the canonical WIP; no local Lean build.
…orrection for rc2 Under rc2 the root-namespace add_apply/sub_apply/mul_apply no longer apply to matrices; use the Matrix.* versions, as rc2-refresh/1676 and /1677 do for the lower modules of this stack. Proof-only; no statement changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 34406a0)
… feat/detectors-affine-1444 Resolves the add/add conflict in RetentionChannel.lean with main's (cslib-community#1688) version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the RetentionChannel.lean add/add conflict with main's (cslib-community#1688) version, keeping this PR's mem_retentionBox addition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion-1444 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion-1444 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edemaine
left a comment
There was a problem hiding this comment.
reviewer-codex-edemaine: APPROVE at eb8ee8e4f85afe9ce30842a0908d791af57805c4 (independent agent integration-delta review, not human approval). Reuse my whole-layer correctness/API/source review and the refreshed #1698/#1691 parent reviews.
FrameRetention and its concrete tests are byte-identical to the reviewed revision. Comparing the full tree with the two reviewed parent heads finds exactly their union plus those two files and one aggregate import; no unexpected source/blob/mode changes. The earlier Matrix proof repairs are supplied by the parents. All frame-density hypotheses, conditioning losses, detector constants and scope remain unchanged; no curated-abstract impact.
git diff --check passes. Exact-head Lean CI 41034759591 was running when inspected; title CI passes. Author reports all six local gates and hygiene passing. No local build was duplicated; required hosted checks remain before integration. Live head rechecked before posting.
|
|
Parents cslib-community#1691, cslib-community#1696 and cslib-community#1698 have landed. The add/add conflicts in KthLargest.lean and ColumnDetector.lean take main's later versions; this PR's own layer does not touch them. Complexity.lean regenerated by mk_all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015DLg8kwPs8yi7ePPdtoNvG
|
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015DLg8kwPs8yi7ePPdtoNvG
|
|
Part of #1682 (Phase 3 of #1444).
This PR proves items 2 and 3 of Lemma 3.1 (the rank-channel estimates) for the frame-conditioned channels of #1691: the pair bound and the lower tail. It then shows that these channels satisfy the detector interface
RetentionChannelsof #1696, mirroring #1698's instance for the unconditioned affine channels. Item 1, the exact marginal, is already #1691'sdens_eval_mem_retentionBox. Item 4, invariance under affine substitution, is in #1711, which depends only on #1691. This is new mathematics for the library, following the ideas of the preprint.Stack. The branch merges, without rebasing:
feat/detectors-affine-1444@0fd3e72a8, which carries feat(Foundations): pilot and probabilistic column detectors for one hierarchy stage #1696 and feat(Probability): affine retention channels over a prime field #1688);feat/conditional-averaging-1444@53de6a3f6);origin/main@0ba8b2e59, the rc2 bump;rc2-refresh/1677@19cf8a82f, which carries the rc2 fixes for feat(Foundations): correction factorization and reward transport for transition matrices #1676 and feat(Foundations): factorial truncation of cutoff corrections #1677.34406a084fixes two further rc2 breakages in the stack: it qualifiesMatrix.add_apply,Matrix.sub_applyandMatrix.mul_applyin #1683'sCopyStage.leanand #1696'sPilotCorrection.lean. It changes proofs only and belongs with the rc2 refresh of those PRs. Please review only this PR's own commit,e6b85378c.Main statements (
Complexity/Probability/PairwiseIndependentHash/FrameRetention.lean, 355 lines). HereFrame P H randFrameChannel P H r = (Fin H → ZMod P) × Frame P H rcome from #1691, andA.eval y = a₀ + ∑ j, y ^ (j + 1) • F j.All results except
frameDensity,frameDensity_nonneg,polyChannelEvalanddens_frameChannel_leassume[Fact P.Prime].Source → Lean ([OpenAI2026LogSpace], revision
7ef5c0cc, printed pages; only the preprint text was read, and no OAI Lean code was opened):frameDensity,card_frame(Mathlib'scard_linearIndependent),frameDensity_eq_prod,frameDensity_pos, andone_div_two_le_frameDensity(c_fr = 1/2forr < H, i.e.h₀ < H + 1; the paper has2 ≤ h₀ < H)polyChannelEval,isPairwiseIndependentHash_polyChannelEval(r ≥ 1, viaisPairwiseIndependentHash_add_of_surjective)dens_frameChannel_le(any event),dens_eval_mem_and_eval_mem_le(two ratesL₁,L₂)4/(qπ_a), then the factor1/q_frdens_two_mul_card_filter_eval_mem_lt_leretentionChannels_frameRetains(C_pair = 2^H/q_fr,C_var = 4/q_fr),retentionChannels_frameRetains_of_lt(2^(H+1),8; constants depending onHonly, as Lemma 3.1 states)Differences from the source. The statements are ours. Item 2 is stated as a joint probability rather than a conditional one:
Pr{both} ≤ π_u π_a / q_fr, and dividing byπ_ugives the paper'sPr{X_a(y) | X_u(x)} ≤ π_a/q_fr ≤ (C_π/q_fr) a. The paper's lower boundq_fr ≥ c_fris made explicit as1 - P^r/P^H ≥ 1/2. It is proved by induction, using(1 - u)² ≥ 1 - P u.Tests (
ComplexityTests/FrameRetention.lean, 101 lines):frameDensity 3 2 1 = 8/9,Fintype.card (Frame 3 2 2) = 48, and1/2 ≤ frameDensity 5 3 2.𝔽₃withH = r = 1and box{0, 1}, adecidecount shows that exactly 2 of the 6 frame-conditioned channels retain both0and1. The joint probability is therefore1/3 ≠ (2/3)², so frame conditioning breaks pairwise independence. The pair bound evaluates to(2/3)²/q_fr = 2/3and covers it.retentionChannels_frameRetains_of_lton𝔽₅, withH = 2,r = 1anda = 1/2, feedslt_card_of_lt_columnDetector: withJ = 20no column is ever flagged. This is the same check as feat(Probability): affine retention channels satisfy the detector interface #1698's test, now for frame channels.Validation at
e6b85378c, all passing locally:scripts/check.sh(all six gates);python3 scripts/check_hygiene.py --check-simp;python3 scripts/check_changed_files.py --base origin/main;python3 scripts/check_tidy_surface.py(baseline unchanged);python3 scripts/parallel_defs.py(no candidates).ROADMAP is untouched. There is no catalogue or abstract impact.
Remaining.
L^sfors > 2, and validated words for the product walk.Review. Please arrange a non-author review of
e6b85378c: correctness, API (naming, placement, theframeRetainsabbreviation) and the source mapping above.Integration refresh (2026-10-30): merged the refreshed #1698 head
5a6439697(contains #1696915171329) and the refreshed #1691 headae99b483bateb8ee8e4f, no rebase, no conflicts. The earlier rc2 fix34406a084is now redundant (its CopyStage half is on main, its PilotCorrection half is in #1696), so against the merged parents this PR only addsFrameRetention.lean, its test and the import line; that own-layer diff is unchanged. All sixscripts/check.shgates,check_hygiene.py --check-simp,check_changed_files.py --base origin/mainandcheck_tidy_surface.pypass ateb8ee8e4f.AI assistance: written by Claude Code (Claude Opus 5.5) as worker
worker-crei-dc679113-1444p3e, for coordinatorcoordinator-crei-dc679113.🤖 Generated with Claude Code