Skip to content

fix(secret-scan): allowlist the plaintext-scan test fixtures - #77

Merged
nkg merged 1 commit into
mainfrom
fix/betterleaks-test-fixture-allowlist
Sep 8, 2026
Merged

fix(secret-scan): allowlist the plaintext-scan test fixtures#77
nkg merged 1 commit into
mainfrom
fix/betterleaks-test-fixture-allowlist

Conversation

@nkg

@nkg nkg commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Summary

The pre-push betterleaks hook has been blocking every push from a clean tree, on any branch, regardless of what it touched. Three findings, all in tests/plaintext-scan.test.sh, introduced in 18d7bed (#75) and sitting on main:

rule line fixture
generic-api-key 55 check flag "base64 private key"
generic-password 68 check clean "sops ciphertext"
generic-password 70 check clean "enum with comment"

All low confidence, and all false positives in the only sense that matters: that file is a corpus of deliberately secret-shaped strings. It feeds fixtures to the sops-audit plaintext scanner and asserts which ones get caught, so roughly half are true-positives by construction. The default rules are behaving correctly; the file is just the wrong thing to point them at.

The only way past it was --no-verify — which disarms the hook for the whole push, not just the known lines. That's the wrong trade to make routine.

Approach

Adds a root .betterleaks.toml, auto-discovered by both the lefthook hooks and secret-scan.yml (whose config-path input already defaults to empty and documents exactly this).

Two choices worth calling out:

Scoped to the single path, not tests/**. The rest of the tree — other test files included — stays covered.

[extend] useDefault = true. Without it, a config file replaces the built-in rules rather than adding to them. The scan would then find nothing and report success, which is precisely the fail-quiet direction the sops-audit allowlist and its test suite exist to guard against. Doing that here, in the secret-scanning config, would be an unusually bad place to get it wrong.

I considered fingerprint entries in .betterleaksignore instead, which would be more surgical. Rejected: fingerprints are pinned to commit:path:rule:line, so every new fixture added to this file — an expected, routine change, since the fixtures are what guard the allowlist — would break the push until someone regenerated them. That papercut would eventually get "fixed" by someone reaching for a blanket path exclusion anyway.

Test plan

  • Full-history betterleaks git scan: clean (was 3 findings).
  • betterleaks config check: 417 rules, matching the default set — confirms extend took effect and nothing was silently dropped.
  • Detection still live: a probe file with a non-example secret in a non-allowlisted path is still caught, firing both stripe-access-token and generic-password. (My first probe used the canonical AWS documentation example key and was not flagged — betterleaks appears to know it. Worth knowing if you ever write a detection test here.)
  • tests/plaintext-scan.test.sh: 21/21 passing, unchanged.
  • This branch pushed with hooks enabled and no --no-verify — all five pre-push hooks green. That's the actual proof.

Notes

Unrelated observation, deliberately not fixed here: a betterleaks dir . scan flags examples/README.md:346 (GRAFANA_ADMIN_PASSWORD=ci-validate, a documentation placeholder). It doesn't affect anything today — both the hooks and secret-scan.yml use betterleaks git, never dir, and the git scan doesn't surface it. Mentioning it only so it isn't a surprise if a dir scan is ever added.

Also: .betterleaks.toml is only auto-discovered when the scan target is the repo root — betterleaks dir tests/ looks for tests/.betterleaks.toml and silently falls back to defaults. CI and the hooks always scan from the root, so this is fine as-is, but it's a sharp edge if anyone adds a narrower scan later.

Stacked note: this and #76 both add a ### Fixed block under ## [Unreleased], so whichever merges second will need a trivial CHANGELOG conflict resolved.

🤖 Generated with Claude Code

https://claude.ai/code/session_016vhsMyme4mZsF49j38WXQB

tests/plaintext-scan.test.sh is a corpus of deliberately secret-shaped
strings — it feeds fixtures to the sops-audit plaintext scanner and asserts
which are caught, so half are true-positives by design. The default rules
flagged three of them (generic-password x2, generic-api-key, all low
confidence) on every full-history scan.

That blocked the pre-push betterleaks hook for everyone, from a clean tree,
on branches that had touched nothing related. The only way past it was
--no-verify, which disarms the hook entirely for that push.

Adds a root .betterleaks.toml, auto-discovered by both the lefthook hooks
and secret-scan.yml (config-path defaults to empty). Two deliberate choices:

- Scoped to the single path, not tests/**. The rest of the tree, other test
  files included, stays covered.
- [extend] useDefault = true. Without it the file REPLACES the built-in
  rules rather than adding to them, disarming the scan while it continues
  to report success — the same fail-quiet direction the sops-audit
  allowlist guards against.

Verified: full-history scan clean; config check reports 417 rules, matching
the default set; a probe secret in a non-allowlisted path is still caught
(stripe-access-token and generic-password both fire); plaintext-scan tests
still 21/21.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vhsMyme4mZsF49j38WXQB
@nkg
nkg force-pushed the fix/betterleaks-test-fixture-allowlist branch from fa8a759 to 3f2d8c2 Compare September 8, 2026 19:42
@nkg
nkg merged commit 3d02124 into main Sep 8, 2026
12 checks passed
@nkg
nkg deleted the fix/betterleaks-test-fixture-allowlist branch September 8, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant