[initial-d]

Summary

N-gram rows with identical complete bag-incidence/multiplicity signatures receive identical updates under the demonstrated sparse-SGD rules, although independent initialization can leave final vectors dissimilar. This adds an opt-in compact parameterization with a group-size gradient correction, certified native export/query plans, and rejection witnesses for unsupported bags. It preserves demonstrated updates and certified aggregates in real arithmetic; individual source vectors can change. The ordinary original-ID/byte-preserving packer remains a separate contract.

The public API works through ngram_store and historical meft_memory. Native export adds optional raw blocks, sequential reordered-ID writes, and release of dense initialization after grouped construction. Batched integer certification now speeds native-plan construction while producing identical plan arrays.

The final evaluation compares actual trained rows against FAISS scalar quantization, PQ, OPQ, residual quantization, full-vector clustering and encoded-byte dedup. Within a declared quality budget, certified groups add complete-model capacity savings beyond the smallest qualifying direct codec. Full-domain fallback, precomputed known-word inference and unfavorable timing/quality results remain visible.

Evidence and limits

• Training and method: 18 matched CPU SGNS pairs across orders 4/8/12, three seeds and two gradient conventions. Actual update cosine exceeds 0.999999999999; final-vector cosine depends on initialization and the gradient rule. Known-word objective relative differences are at most 2.06e-8 in a post-run audit. The certificate characterizes equal member counts as the maximal weight-independent bag domain for mean projection; rejected queries receive a sparse algebraic counterexample. These results do not establish semantic/PPL/LLM quality or first-of-kind novelty.
• Mature compression controls: nine cached 128-dimensional SGNS checkpoints, 19 codecs × three contracts = 513 base candidates. Another 171 actual-code dedup audits yield 64 smaller candidates and 107 flat-preferred results, giving 577 scored variants with no final failures. Select on validation at loss increases of 0.01%/0.1%/0.5%, discrimination decline ≤0.5 percentage point and aggregate relative L2 ≤5%; audit selected models on test.
• Fixed-budget capacity: at the 0.1% loss budget, median complete savings versus the smallest qualifying direct codec are 11.27% / 33.21% / 34.54% at orders 4/8/12. All select SQ6; grouped reconstruction is less accurate but within the common bound. All selected test point estimates pass. The grid extension is exploratory, and some bootstrap intervals cross the stricter 0.01% loss limit.
• Strong negative controls: full-domain hybrid with every original nonsingleton FP32 fallback row is larger than direct SQ6. Certified nonempty OOV coverage is only 97/512 or 54/512. Known-word precomputation costs 1,240,108 bytes and about 0.029 ms per 512-word gather, with a narrower inference-only contract. Even 2,048-center clustering violates the 5% aggregate-error gate. Neither arbitrary-query nor equal-accuracy dominance over FAISS is claimed.
• Query optimization: grouped SQ6 precompiled replay is faster on these warm traces; scalar certificate compilation plus FAISS query is slower than direct. Separately measured batch certification is 7.3–7.8× faster and native-plan construction 3.1–3.4× faster, with every plan array byte-identical. Compilation gains are not an end-to-end serving claim or a retiming of the frozen FAISS workers.
• Large-table and original algorithm: the existing 1,855,907 × 256 updated-row subset has 360,088 groups; it was not retrained as a large grouped model. Certified raw layout plus keys/IDs/certificate/plan costs 436,343,333 bytes versus 1,824,772,214 source-native bytes. Raw certified queries improve 529.9 → 15.82 ms with 5.62% more complete bytes; raw mmap and known-word precomputation remain faster. Independent DATrie mark-compact controls add 11.1–11.8% complete savings after canonical projection; byte dedup alone adds 22.6–36.6%, so overlapping gains are not attributed exclusively to DATrie. Large original-byte incidence reordering saves only 0.06674%.

Independently gated Engram heads, arbitrary LLM memory fields, large grouped retraining, construction under a fixed RAM budget and independent corpora remain outside the demonstrated result. The current large Python builder peaks near 5.4 GiB RSS.

Evidence artifacts

• Final FAISS report, four curves, raw metrics, uncertainty, provenance and commands.
• Matched co-training, large capacity and independent DATrie report.
• API/optimizer/query assumptions and contribution/prior-work boundaries.

Verification

• Full Linux suite at e6d2870: python -m unittest discover -s tests — 165 tests, 17 skipped, successful, including FAISS CPU and PyTorch cases. Full log is published.
• Final report audit verifies all 513 base row checks, 171 dedup original-ID checks, unique variants, 81 selections, input/source/report hashes, JSON and local links. Four figures were generated and visually inspected; shell syntax, Python compile checks and git diff --check pass.
• Local FAISS module suite after the figure-layout change: six tests, one skipped because FAISS is unavailable locally, successful.
• Installed-wheel public API example and earlier native/planner checks remain source-linked in the co-training report.
• Latest-head GitHub CI at f9994ca: both PR and push workflows completed successfully, covering CPU PyTorch/FAISS, native/FastText sanitizers, legacy compatibility, Engram/TinyEngram integration and wheels.

Benchmark context

Cached TinyShakespeare finite SGNS models and the separately scoped enwik8 updated-row export; no random performance vectors or corpus downloads. AMD EPYC 7742 shared CPU, Linux x86-64, Python 3.11.15, PyTorch 2.11.0+cu130, NumPy 2.2.6, FAISS CPU 1.15.0. Query threads one; base fits one, extended fits four. Warm OS pages and overlapping phases limit speed interpretation. CUDA_VISIBLE_DEVICES=''; no GPU process inspected or stopped. Phase commits, exact commands, source hashes, failed/superseded attempts and full-cost controls are retained. No private text, vocabulary, weights, credentials or internal hostnames are included.

Type of change

• Opt-in parameterization and certified native API
• Query-plan/construction and tooling improvements
• Trained-row benchmark and reproducibility artifacts
• C++ behavior change