Repository navigation
test(oracle): read empty-bus NACKs on F460/F461/MyHomeServer1 by verdict - #766
Merged
GreenGrassBlueOcean merged 1 commit intoOct 21, 2026
Merged
Conversation
On F460/F461 (3-digit PIC protocol) and MyHomeServer1 2.82.06 (2-digit v8 PIC mode), scsserver reports a bus write as delivered only when a device acknowledges it ($00800 / $08). The oracle's full suites run on an empty bus, so these gateways reply NACK to almost every frame they forward. Live F461 2.0.11 traces ACK all of those frames, and the oracle's F461 plant run ACKs them once devices answer. - test_golden_corpus_against_firmware_oracle now counts a frame as refused on those three gateways only when its verdict is not "out". - Remove the 21 KNOWN_GATEWAY_DISCREPANCIES entries that only recorded those empty-bus NACKs; the 25 remaining entries are real refusals. - test_dimmer_level_cross_gateway_behavior no longer describes the MyHomeServer1 NACK as a gateway divergence. - New test fails once the oracle stops recording empty-bus NACKs, so the special case is removed then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced Oct 21, 2026
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #745
What is the problem
In
tests/golden/firmware_oracle.json, F460, F461 and MyHomeServer1 reply NACK to 228 of the 309 frames they put on the bus (74 %). For the F454 or MH200N it is about 15 %. Those NACKs come from how the oracle is run, not from the firmware refusing the frames:scsserverreports a bus write as delivered only when the PIC answers$00800, meaning a device acknowledged the telegram. MyHomeServer1 2.82.06 runs its 2-digit PIC protocol in "v8" mode, whereWrite()rejects$19/$00and requires$08. Without it,scsserverretries three times and the command session replies NACK after about 2 s.fullsuites run against a simulated PIC with no devices behind it, which always answers$019/$000. Older gateways (F453/F454/MH200/MH200N/H4684) ACK on transmission, so their replies don't depend on what is on the bus.Evidence:
README.md(around line 214) andfindings/pic-inbound-protocol.mddocument this from thescsserverbinaries.results/F461/020011/oracle/plant-f461-bench/) ACKs*1*0*0##and*#18*51*#1200#1*1##once simulated devices answer$00800. It NACKs only the addresses deliberately left empty.*1*1*0##,*#2*0##and*#18*51*#1200#1*0##, all recorded asnack/outin the oracle. Every frame the oracle saw gooutwas ACKed live, and everysilentone was NACKed (test_f461_live_trace_agrees_with_oracle, from fix(oracle): verify OWNd's own gateway profiles and check F461 live traces against the oracle (#735) #745).MyHOME's tests read these NACKs as gateway behaviour:
KNOWN_GATEWAY_DISCREPANCIESentries recorded empty-bus NACKs as expected F460/F461/MyHomeServer1 behaviour (e.g. "the F461 NACKs*1*1*0##", which the real gateway ACKs).test_golden_corpus_against_firmware_oracletreated every NACK as a refusal, so it could only pass by listing those entries.test_dimmer_level_cross_gateway_behaviordescribed the MyHomeServer1 NACK on*1*0#1*31##as a divergence from the MH200N.How is it solved
All changes are in
tests/test_firmware_oracle_conformance.py:EMPTY_BUS_NACK_GATEWAYSand_refused(): on F460/F461/MyHomeServer1 a frame counts as refused only when its verdict is notout. Everywhere else,reply == "nack"still counts.KNOWN_GATEWAY_DISCREPANCIES. The 25 that remain are frames the translator dropped (silent), plus the existing MH202light.cmd.off.bus.25entry.test_dimmer_level_cross_gateway_behaviornow asserts that both gateways forward the frame and neither refuses it, and its docstring explains the reply difference.test_empty_bus_gateways_still_nack_forwarded_framesfails once the oracle stops recording empty-bus NACKs for a gateway. When that happens, remove the gateway from the set. The upstream fix (re-running those suites with devices that acknowledge, or marking the replies in the export) is being prepared in OpenWebNet-HA/own-firmware-oracle.How is it tested
reply == "nack"rule and the shorter discrepancy list,test_golden_corpus_against_firmware_oraclefails on exactly the removed F460/F461/MyHomeServer1 entries.ruff,mypy,scripts/verify_ha_standards.pyandscripts/sync_documentation.py --checkpass.Where are the traces
tests/fixtures/traces/f461_live_2026-10-18/myhome_trace_F461_live_2026-10-18.jsontests/fixtures/traces/f461_who14_live_2026-10-18/myhome_trace_F461_who14_live_2026-10-18.jsonNo fixtures are added or changed.
What does the oracle say
*1*1*0##/*1*0*0##(general light)$00800)*#2*0##(general automation status)*#18*51*#1200#1*0##/*1##(energy)$00800)*#1*08##,*#1*25##,*#1*50##,*#1*99##(no device)$019)🤖 Generated with Claude Code