fix(secret-scan): allowlist the plaintext-scan test fixtures - #77
Merged
Conversation
tests/plaintext-scan.test.sh is a corpus of deliberately secret-shaped strings — it feeds fixtures to the sops-audit plaintext scanner and asserts which are caught, so half are true-positives by design. The default rules flagged three of them (generic-password x2, generic-api-key, all low confidence) on every full-history scan. That blocked the pre-push betterleaks hook for everyone, from a clean tree, on branches that had touched nothing related. The only way past it was --no-verify, which disarms the hook entirely for that push. Adds a root .betterleaks.toml, auto-discovered by both the lefthook hooks and secret-scan.yml (config-path defaults to empty). Two deliberate choices: - Scoped to the single path, not tests/**. The rest of the tree, other test files included, stays covered. - [extend] useDefault = true. Without it the file REPLACES the built-in rules rather than adding to them, disarming the scan while it continues to report success — the same fail-quiet direction the sops-audit allowlist guards against. Verified: full-history scan clean; config check reports 417 rules, matching the default set; a probe secret in a non-allowlisted path is still caught (stripe-access-token and generic-password both fire); plaintext-scan tests still 21/21. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vhsMyme4mZsF49j38WXQB
nkg
force-pushed
the
fix/betterleaks-test-fixture-allowlist
branch
from
September 8, 2026 19:42
fa8a759 to
3f2d8c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
pre-pushbetterleaks hook has been blocking every push from a clean tree, on any branch, regardless of what it touched. Three findings, all intests/plaintext-scan.test.sh, introduced in 18d7bed (#75) and sitting onmain:generic-api-keycheck flag "base64 private key"generic-passwordcheck clean "sops ciphertext"generic-passwordcheck clean "enum with comment"All low confidence, and all false positives in the only sense that matters: that file is a corpus of deliberately secret-shaped strings. It feeds fixtures to the sops-audit plaintext scanner and asserts which ones get caught, so roughly half are true-positives by construction. The default rules are behaving correctly; the file is just the wrong thing to point them at.
The only way past it was
--no-verify— which disarms the hook for the whole push, not just the known lines. That's the wrong trade to make routine.Approach
Adds a root
.betterleaks.toml, auto-discovered by both the lefthook hooks andsecret-scan.yml(whoseconfig-pathinput already defaults to empty and documents exactly this).Two choices worth calling out:
Scoped to the single path, not
tests/**. The rest of the tree — other test files included — stays covered.[extend] useDefault = true. Without it, a config file replaces the built-in rules rather than adding to them. The scan would then find nothing and report success, which is precisely the fail-quiet direction the sops-audit allowlist and its test suite exist to guard against. Doing that here, in the secret-scanning config, would be an unusually bad place to get it wrong.I considered fingerprint entries in
.betterleaksignoreinstead, which would be more surgical. Rejected: fingerprints are pinned tocommit:path:rule:line, so every new fixture added to this file — an expected, routine change, since the fixtures are what guard the allowlist — would break the push until someone regenerated them. That papercut would eventually get "fixed" by someone reaching for a blanket path exclusion anyway.Test plan
betterleaks gitscan: clean (was 3 findings).betterleaks config check: 417 rules, matching the default set — confirmsextendtook effect and nothing was silently dropped.stripe-access-tokenandgeneric-password. (My first probe used the canonical AWS documentation example key and was not flagged — betterleaks appears to know it. Worth knowing if you ever write a detection test here.)tests/plaintext-scan.test.sh: 21/21 passing, unchanged.--no-verify— all five pre-push hooks green. That's the actual proof.Notes
Unrelated observation, deliberately not fixed here: a
betterleaks dir .scan flagsexamples/README.md:346(GRAFANA_ADMIN_PASSWORD=ci-validate, a documentation placeholder). It doesn't affect anything today — both the hooks andsecret-scan.ymlusebetterleaks git, neverdir, and the git scan doesn't surface it. Mentioning it only so it isn't a surprise if adirscan is ever added.Also:
.betterleaks.tomlis only auto-discovered when the scan target is the repo root —betterleaks dir tests/looks fortests/.betterleaks.tomland silently falls back to defaults. CI and the hooks always scan from the root, so this is fine as-is, but it's a sharp edge if anyone adds a narrower scan later.Stacked note: this and #76 both add a
### Fixedblock under## [Unreleased], so whichever merges second will need a trivial CHANGELOG conflict resolved.🤖 Generated with Claude Code
https://claude.ai/code/session_016vhsMyme4mZsF49j38WXQB