Skip to content

fix: Chapter 5's claims match the data it ships - #128

Merged
qmohsu merged 1 commit into
mainfrom
claude/ch5-claims-match-shipped-data
Aug 31, 2026
Merged

qmohsu merged 1 commit into
mainfrom
claude/ch5-claims-match-shipped-data

Conversation

@qmohsu

@qmohsu qmohsu commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Eight audit findings, each re-measured before it was fixed. No dataset byte moved.

1. The −0.34 mechanism belonged to data that does not ship

report_what_a_repeat_survey_buys explained MAP's loss to 1-NN by an anti-correlation between the spatial mismatch and the per-visit σ, and explicitly called the estimation-noise diagnosis "wrong". Measured on the shipped multisample survey that correlation is +0.0156 — zero, because the fast fading is one constant for the whole building.

The estimation-noise diagnosis is right, and the demo now runs the proof instead of asserting it. A new oracle row substitutes the true constant σ into the fitted model:

  min_std     sigma range    MAP!=NN   max wt  eff RPs      MAP       NN
      0.5   0.50 -  3.02     44/200   0.9873     1.03   3.72 m   3.16 m
      ...
   oracle   1.50 -  1.50      0/200   0.9788     1.04   3.16 m   3.16 m

The anti-correlation survives as a labelled conditional about a non-shipped variant. Its dose-response column needed dense and sparse repeat surveys that do not exist, so it is gone; the spatial-mismatch column is shipped-verifiable and re-measured at 0.54 / 1.43 / 2.98 dB.

2. Two stale constants

"2.27 m quantisation floor" matches nothing measurable. sqrt(2 s²/12) = 2.041 for s = 5; sampling 200k uniform positions gives 2.043 rms and 1.91 mean; clean-map NN achieves 2.176. RMS is now the named statistic, because everything compared against it is an RMSE, and the mean is labelled as the mean in the dense/sparse survey-effort tables. The "3.39 m" headline row re-measures to 3.01 m (20 seeds: 2.95–3.44).

3. The sparse grid is 6×6

arange(0, 55, 10) has six nodes. 36 RPs/floor and 108 total, not 25/75.

4. example_pattern_recognition demonstrated nothing

diag(Z'Z) is 3.6e5–4.3e5, so every swept λ (0, 0.1, 1, 10) was under 3e-5 of the data term — train RMSE printed 6.09 four times under "Regularization prevents overfitting", with train below test at every λ.

Now: λ ∈ {0, 10, 100, 1e3, 1e4} (measured, the knob bites near 1e2); symlog x so λ=0 is in the panels; and a training-set-size sweep supplying the overfitting the 85-RP split lacks — at 9 training points for 8 features λ=0 fits exactly and scores 167 m, λ=100 returns 10.5 m.

5. Coincident series erased each other

NN ≡ MAP and PostMean ≡ PostMean(k=10), so panels B/F/I showed four series for six legend entries. One shared stroke and marker table now covers B, E, F and I; Key Insight 6 names both causes with both gaps computed.

6. example_probabilistic never stated its degeneracy

Prints sigma_summary() per model, captions the flat MAP line, dashes apart the three identical MAP CDFs, and the x=y reference spans both axes instead of being the zero-length segment the legend advertised.

7. Two test files asserted nothing

Seven test_-named functions with zero asserts, printing [OK]/[FAIL] and returning True; pytest passed all seven regardless. Now 13 assertions, each verified by mutation. One overturned the old file's own printed claim: widening an RP's σ widens its basin rather than reducing its influence — the if guarding that claim had silently never fired.

8. Minors

2.86x → 1.09x and 30x → 60.5x (operations, sourced); scenario labels naming 4 dB shadowing where the unlabelled term is 1.5 dB fast fading; panel G's n=1 category; the σ=0 row in the classification figure; test_shadowing_field's "never exceeded 0.63" measured at 0.658.

Test plan

  • tests/core/fingerprinting tests/ch5_fingerprinting — 287 passed
  • tests/test_datasets_reproduce_from_their_recipe.py — 43 passed, no data byte moved
  • tests/docs — 880 passed, 1 skipped
  • validators, semantic, lint, mypy, black, doctest, repo-conventions — 845 passed in the consolidated run
  • all six ch5 examples run; four figures regenerated and opened
  • MAP≡NN invariants: single-sample 0/200 differ, multisample 44/200 (22%) at min_std 0.5, effective RP count 1.03–1.79 (posterior non-delta)

🤖 Generated with Claude Code

An audit found eight places where Chapter 5 asserts something its own
datasets do not support. Each is fixed by measuring, and where possible by
making the demo compute the claim instead of printing it.

**The -0.34 mechanism was scoped to data that does not ship.** The repeat-survey
section explained MAP's loss to 1-NN by an anti-correlation between the spatial
mismatch and the per-visit sigma, and explicitly called the estimation-noise
diagnosis "wrong". On the shipped multisample survey that correlation is
+0.0156 -- zero, because the fast fading is one constant for the whole building
and a constant has nothing to anti-correlate with. The estimation-noise
diagnosis is right there, and the example now proves it rather than asserting
it: an `oracle` row substitutes the true constant sigma (1.5 dB) into the fitted
model and MAP differs from 1-NN on 0/200 queries at an identical 3.16 m. The
anti-correlation is kept as a labelled conditional about a non-shipped variant;
its dose-response column, which needed dense and sparse repeat surveys that do
not exist, is gone. The spatial-mismatch column is shipped-verifiable and stays,
re-measured at 0.54 / 1.43 / 2.98 dB.

**Two stale constants.** "2.27 m quantisation floor" matches nothing: the
formula sqrt(2 s^2 / 12) is 2.041 for s = 5, sampling gives 2.043 rms and 1.91
mean, and a clean-map NN scores 2.176. RMS is now the named statistic
everywhere, because everything it is compared against is an RMSE, and the mean
is labelled as the mean where the dense and sparse tables use it. The "3.39 m"
headline row re-measures to 3.01 m.

**The sparse grid is 6x6.** `arange(0, 55, 10)` has six nodes, so the variant is
36 RPs per floor and 108 total, not the 25/75 the grid README claimed twice.

**example_pattern_recognition demonstrated nothing.** diag(Z'Z) is 3.6e5-4.3e5,
so every lambda it swept (0, 0.1, 1, 10) was under 3e-5 of the data term and
train RMSE printed 6.09 four times under "Regularization prevents overfitting".
The sweep now runs 0 to 1e4, where the knob visibly bites; the two log-x panels
are symlog so lambda=0 is in them; and a training-set-size sweep supplies the
overfitting the 85-RP split does not have -- at 9 training points for 8 features
the unregularised fit is exact and scores 167 m, where lambda=100 returns 10.5 m.

**Coincident series no longer erase each other.** NN is MAP exactly and the two
posterior means agree to five figures, so panels B, F and I drew four series for
six legend entries. One shared stroke/marker table now covers B, E, F and I, and
Key Insight 6 names both causes with both gaps computed.

**example_probabilistic never stated the degeneracy it displayed.** It prints
`sigma_summary()` per model, captions the flat MAP line, dashes apart the three
identical MAP CDFs, and its x=y reference line spans both axes instead of being
the zero-length segment the legend advertised.

**Two test files asserted nothing.** `test_topk_posterior_mean.py` and
`test_multisamples.py` held seven `test_`-named functions with zero assertions
that printed [OK]/[FAIL] and returned True; pytest passed all seven whatever the
library did. They are 13 real assertions now, each verified to fail under a
mutation. One of them overturned the file's own printed claim: widening a
reference point's sigma *widens* its basin rather than reducing its influence,
which the old `if` had silently never confirmed.

Minors: the 2.86x and 30x speedup claims (measured 1.09x and 60.5x operations),
the scenario labels naming 4 dB of shadowing where the unlabelled term is 1.5 dB
of fast fading, panel G's n=1 category, the sigma=0 row in the classification
figure, and a seed bound in test_shadowing_field measured at 0.658 not 0.63.

No dataset byte moved; the four regenerated figures are the four whose plotting
changed, and every RMSE, median and P90 in the comparison example is unchanged.
@qmohsu
qmohsu merged commit 4b9adb0 into main Aug 31, 2026
1 check passed
@qmohsu
qmohsu deleted the claude/ch5-claims-match-shipped-data branch August 31, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant