Repository navigation
data_validation: check mapping rows that cite published test records - #106
Open
astrogilda wants to merge 4 commits into
Open
astrogilda wants to merge 4 commits into
astrogilda wants to merge 4 commits into
Conversation
Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Add evidence_mapping_check, a shared check that replays the corpus a framework_alignments row cites with its reference verifier, checks the cited requirement exists and tests the row's control, and runs from run_all_checks. Allow test_records and OWASP MCP Top 10 in the risk assessment schema. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
…ange The schema checksum recorded in croissant.json no longer matched schema.json. Regenerated with data_validation/croissant/build_croissant.py. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
validators/evidence_mapping_check.py, the check I promised in the channel on 10 Oct, and it runs fromrun_all_checks.py.A mapping row can carry an optional
test_recordsfield. It names a corpus of signed accept and reject records inagent-evidence-vectors(pinned at 0.24.0), and optionally one requirement in it. The check fails when the corpus is missing from the pinned version, or the requirement is absent, withdrawn or silent on the row's control. It also fails when the corpus does not replay clean under the package's reference verifier. Rows markedmissingornot_applicableget a warning.The check is opt-in. It acts only once mapping rows carry the field. No row cites an MCP Top 10 control yet, so today it passes on every file and changes nothing.
The existing
pipentry independabot.ymlcoversdata_validation/, so each release of the package opens a bump PR that this check tests. Ruff, mypy and the suite pass.