Found in epic #324 Phase 15 (#351).
The harness fits pamica and the reference from independent random starts at one seed. So its log-likelihood gap is one draw from the start-to-start spread:
- the reference's own final LL has a standard deviation of 2.6e-4 over 8 seeds at the harness settings;
- the harness gap moved from 2.9e-5 to 2.7e-4 when the epic changed the initial draw.
The update-rule comparison is the shared-start one: the reference loaded with pamica's initial state gives an LL gap of 1.6e-6 and a correlation of 0.99999993.
Proposal: have validate_implementations.py also report a shared-start comparison through native_oracle.run_seeded_reference, the reference's own seed spread, or both, next to its independent-start figure, so the output carries its own context.
Scripts and records: .context/issue-351/harness_gap.py, raw/harness_gap_head.json.
Found in epic #324 Phase 15 (#351).
The harness fits pamica and the reference from independent random starts at one seed. So its log-likelihood gap is one draw from the start-to-start spread:
The update-rule comparison is the shared-start one: the reference loaded with pamica's initial state gives an LL gap of 1.6e-6 and a correlation of 0.99999993.
Proposal: have
validate_implementations.pyalso report a shared-start comparison throughnative_oracle.run_seeded_reference, the reference's own seed spread, or both, next to its independent-start figure, so the output carries its own context.Scripts and records:
.context/issue-351/harness_gap.py,raw/harness_gap_head.json.