feat: close out the scanner improvement backlog - #115
Merged
Merged
Conversation
Implements the five items tracked in #107: - GitHub classic tokens (ghp_/gho_/ghu_/ghs_/ghr_) are validated against their built-in CRC32 checksum: base62 over the 30 random characters, zero-padded to six. Verified against a live token; GitHub's published example token is fabricated and does not self-validate. Non-36-character bodies pass unchecked so a future format is not silently dropped. The checksum-valid test fixture is assembled with concat! because GitHub push protection validates the same checksum and rejects the contiguous literal as a live token. - UTF-16 files and staged blobs that start with a byte-order mark are decoded (both byte orders, lossily) and scanned; Windows-written .env files were previously skipped as binary. BOM-less UTF-16 stays classified as binary: without the mark, distinguishing it from a real binary is guesswork. - Base64 runs of 24+ characters are decoded and the decoded text is scanned once more (no recursion), attributed to the original line. Decoded content must be printable text of at least 16 bytes, so digests and compressed data are rejected cheaply. Inline suppression stays a property of the visible line. - --trusted-detectors (ignore repository detector files) and --no-repo-config (skip .keywatch.toml discovery) split the two behaviors of --no-config-discovery, which remains as their combination so installed hooks keep working unchanged. - --max-file-size <MB> reports larger files as unscannable instead of scanning them, visible to --fail-on-unscannable; staged blobs respect the same cap. No default limit: silently skipping content is worse than a slow scan unless the operator asked for it. Closes #107. Assisted-by: Claude Fable 5 Signed-off-by: PiX <69745008+pixincreate@users.noreply.github.com>
- The reported_by helper and three sibling closures gated matches with accepts_match (whole match), while production gates the captured value with accepts_captures. No fixture was falsely green, but a future fixture could have blessed a detector the scanner never fires. All helpers now mirror the production accept chain exactly. - The unlistable-directory tests print a SKIPPED notice when running as root, where mode 000 cannot construct the scenario, instead of passing silently while verifying nothing. - Existence-only assertions became exact: the multi-secret line pins its full finding list, the Aadhaar test pins exactly one finding, and the PEM header test names SSHPrivateKeyDetector. - README corrections: --fail-on-unscannable documents its strict-mode and --update-baseline limits again, the pre-push hook section says only HIGH/CRITICAL findings block (critical exit mode), the SARIF example writes to a file, and the staged multi-hunk limitation note is restored. - New coverage: an end-to-end pre-push test proves remote..local range semantics with the real binary against real commits (a historical secret outside the range does not block; the same commit inside the range does), and a CLI-level SARIF test validates the written report's version, ruleId, level and location. Assisted-by: Claude Fable 5 Signed-off-by: PiX <69745008+pixincreate@users.noreply.github.com>
Pad the option, exit-code, hook and Action tables to aligned columns, and add notes for the behaviors this branch introduces: UTF-16 files with a byte-order mark are decoded and scanned, base64 runs are decoded and rescanned at their original line, and GitHub tokens are checked against their built-in checksum. Assisted-by: Claude Fable 5 Signed-off-by: PiX <69745008+pixincreate@users.noreply.github.com>
The plain-language rewrite dropped the architecture diagrams and their explanations, keeping only a pointer to docs/architecture. The images carry real orientation value, so the section returns with the diagrams and the explanations rewritten in the same plain style as the rest of the file. Assisted-by: Claude Fable 5 Signed-off-by: PiX <69745008+pixincreate@users.noreply.github.com>
The scan pipeline gained UTF-16 decoding, the file size cap, content validators and the base64 rescan stage, and the trust diagram still described trusted mode as one flag after --trusted-detectors and --no-repo-config split it. Both sources are updated and re-rendered with the pinned D2 v0.7.1; the module diagram is unchanged because nothing moved at that level. The baseline covers the re-rendered SVGs' shifted font data. Assisted-by: Claude Fable 5 Signed-off-by: PiX <69745008+pixincreate@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements all five items from #107 in one change, as requested. Item-by-item:
concat!to be pushable at all..envfiles were previously skipped as binary. BOM-less UTF-16 stays classified as binary on purpose.--trusted-detectorsand--no-repo-configseparate the two behaviors of--no-config-discovery, which remains as their combination so installed hooks keep working unchanged.--max-file-size <MB>— larger files and staged blobs are reported as unscannable rather than scanned; visible to--fail-on-unscannable. No default limit: silently skipping content is worse than a slow scan unless asked for.271 tests pass, clippy clean, self-scan clean, baseline stable under repeated regeneration.
Closes #107.