Repository navigation
fix: Chapter 5's claims match the data it ships - #128
Merged
Merged
Conversation
An audit found eight places where Chapter 5 asserts something its own datasets do not support. Each is fixed by measuring, and where possible by making the demo compute the claim instead of printing it. **The -0.34 mechanism was scoped to data that does not ship.** The repeat-survey section explained MAP's loss to 1-NN by an anti-correlation between the spatial mismatch and the per-visit sigma, and explicitly called the estimation-noise diagnosis "wrong". On the shipped multisample survey that correlation is +0.0156 -- zero, because the fast fading is one constant for the whole building and a constant has nothing to anti-correlate with. The estimation-noise diagnosis is right there, and the example now proves it rather than asserting it: an `oracle` row substitutes the true constant sigma (1.5 dB) into the fitted model and MAP differs from 1-NN on 0/200 queries at an identical 3.16 m. The anti-correlation is kept as a labelled conditional about a non-shipped variant; its dose-response column, which needed dense and sparse repeat surveys that do not exist, is gone. The spatial-mismatch column is shipped-verifiable and stays, re-measured at 0.54 / 1.43 / 2.98 dB. **Two stale constants.** "2.27 m quantisation floor" matches nothing: the formula sqrt(2 s^2 / 12) is 2.041 for s = 5, sampling gives 2.043 rms and 1.91 mean, and a clean-map NN scores 2.176. RMS is now the named statistic everywhere, because everything it is compared against is an RMSE, and the mean is labelled as the mean where the dense and sparse tables use it. The "3.39 m" headline row re-measures to 3.01 m. **The sparse grid is 6x6.** `arange(0, 55, 10)` has six nodes, so the variant is 36 RPs per floor and 108 total, not the 25/75 the grid README claimed twice. **example_pattern_recognition demonstrated nothing.** diag(Z'Z) is 3.6e5-4.3e5, so every lambda it swept (0, 0.1, 1, 10) was under 3e-5 of the data term and train RMSE printed 6.09 four times under "Regularization prevents overfitting". The sweep now runs 0 to 1e4, where the knob visibly bites; the two log-x panels are symlog so lambda=0 is in them; and a training-set-size sweep supplies the overfitting the 85-RP split does not have -- at 9 training points for 8 features the unregularised fit is exact and scores 167 m, where lambda=100 returns 10.5 m. **Coincident series no longer erase each other.** NN is MAP exactly and the two posterior means agree to five figures, so panels B, F and I drew four series for six legend entries. One shared stroke/marker table now covers B, E, F and I, and Key Insight 6 names both causes with both gaps computed. **example_probabilistic never stated the degeneracy it displayed.** It prints `sigma_summary()` per model, captions the flat MAP line, dashes apart the three identical MAP CDFs, and its x=y reference line spans both axes instead of being the zero-length segment the legend advertised. **Two test files asserted nothing.** `test_topk_posterior_mean.py` and `test_multisamples.py` held seven `test_`-named functions with zero assertions that printed [OK]/[FAIL] and returned True; pytest passed all seven whatever the library did. They are 13 real assertions now, each verified to fail under a mutation. One of them overturned the file's own printed claim: widening a reference point's sigma *widens* its basin rather than reducing its influence, which the old `if` had silently never confirmed. Minors: the 2.86x and 30x speedup claims (measured 1.09x and 60.5x operations), the scenario labels naming 4 dB of shadowing where the unlabelled term is 1.5 dB of fast fading, panel G's n=1 category, the sigma=0 row in the classification figure, and a seed bound in test_shadowing_field measured at 0.658 not 0.63. No dataset byte moved; the four regenerated figures are the four whose plotting changed, and every RMSE, median and P90 in the comparison example is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight audit findings, each re-measured before it was fixed. No dataset byte moved.
1. The −0.34 mechanism belonged to data that does not ship
report_what_a_repeat_survey_buysexplained MAP's loss to 1-NN by an anti-correlation between the spatial mismatch and the per-visit σ, and explicitly called the estimation-noise diagnosis "wrong". Measured on the shipped multisample survey that correlation is +0.0156 — zero, because the fast fading is one constant for the whole building.The estimation-noise diagnosis is right, and the demo now runs the proof instead of asserting it. A new
oraclerow substitutes the true constant σ into the fitted model:The anti-correlation survives as a labelled conditional about a non-shipped variant. Its dose-response column needed dense and sparse repeat surveys that do not exist, so it is gone; the spatial-mismatch column is shipped-verifiable and re-measured at 0.54 / 1.43 / 2.98 dB.
2. Two stale constants
"2.27 m quantisation floor" matches nothing measurable.
sqrt(2 s²/12)= 2.041 for s = 5; sampling 200k uniform positions gives 2.043 rms and 1.91 mean; clean-map NN achieves 2.176. RMS is now the named statistic, because everything compared against it is an RMSE, and the mean is labelled as the mean in the dense/sparse survey-effort tables. The "3.39 m" headline row re-measures to 3.01 m (20 seeds: 2.95–3.44).3. The sparse grid is 6×6
arange(0, 55, 10)has six nodes. 36 RPs/floor and 108 total, not 25/75.4.
example_pattern_recognitiondemonstrated nothingdiag(Z'Z)is 3.6e5–4.3e5, so every swept λ (0, 0.1, 1, 10) was under 3e-5 of the data term — train RMSE printed 6.09 four times under "Regularization prevents overfitting", with train below test at every λ.Now: λ ∈ {0, 10, 100, 1e3, 1e4} (measured, the knob bites near 1e2); symlog x so λ=0 is in the panels; and a training-set-size sweep supplying the overfitting the 85-RP split lacks — at 9 training points for 8 features λ=0 fits exactly and scores 167 m, λ=100 returns 10.5 m.
5. Coincident series erased each other
NN ≡ MAP and PostMean ≡ PostMean(k=10), so panels B/F/I showed four series for six legend entries. One shared stroke and marker table now covers B, E, F and I; Key Insight 6 names both causes with both gaps computed.
6.
example_probabilisticnever stated its degeneracyPrints
sigma_summary()per model, captions the flat MAP line, dashes apart the three identical MAP CDFs, and the x=y reference spans both axes instead of being the zero-length segment the legend advertised.7. Two test files asserted nothing
Seven
test_-named functions with zero asserts, printing[OK]/[FAIL]and returningTrue; pytest passed all seven regardless. Now 13 assertions, each verified by mutation. One overturned the old file's own printed claim: widening an RP's σ widens its basin rather than reducing its influence — theifguarding that claim had silently never fired.8. Minors
2.86x → 1.09x and 30x → 60.5x (operations, sourced); scenario labels naming 4 dB shadowing where the unlabelled term is 1.5 dB fast fading; panel G's n=1 category; the σ=0 row in the classification figure;
test_shadowing_field's "never exceeded 0.63" measured at 0.658.Test plan
tests/core/fingerprinting tests/ch5_fingerprinting— 287 passedtests/test_datasets_reproduce_from_their_recipe.py— 43 passed, no data byte movedtests/docs— 880 passed, 1 skipped🤖 Generated with Claude Code