Scan committed history for secrets, not only the working tree

[AlexSkrypnyk]

Summary

Secret scanning runs against the working directory, so it finds a secret that is in the tree right now. It does not find a secret that was committed and then removed in a later commit - the tree at HEAD is clean, the scan passes, and the secret is still retrievable from history by anyone with a clone.

That blind spot is the day-to-day gap, and closing it does not require scanning full history. Scanning the pushed commit range is enough, and it is cheap.

Details

Why a full-history scan is not the answer for routine builds

Full-history scanning is a one-time onboarding activity, not a per-build check. Once history has been scanned and whatever it found has been rotated, re-scanning the same history on every build re-reads the same commits to reach the same answer. It should not be in the routine pipeline.

Separating the three cases makes the split clear:

Case A - a secret committed then removed on the same branch. A developer commits .env, notices, deletes it, commits again and pushes. Both CI providers run gitleaks dir ., which scans the tree as it exists at HEAD and reports nothing. The secret is in the branch history, and after merge it is in the default branch's history permanently. This happens routinely, it is entirely invisible today, and it is caught by scanning the pushed commit range - a handful of commits, not the whole repository.

Case B - detection rules that did not exist when the content was committed. The scanner image is version-bumped by dependency updates, and new releases add rules. A rule added this month can match content that passed unflagged last year. The only way to find those is to re-evaluate history against current rules, and the natural cadence for that is periodic rather than per-build.

Case C - a repository that predates secret scanning. This is the onboarding scan, and it is genuinely once. It needs a documented command and rotation guidance, not a pipeline step.

So the answer to "wouldn't you run this just once" is yes for Case C, no for Case A - and Case A is the reason this issue exists. The original framing conflated them.

Proposed split

1. Commit-range scan on every build - enabled by default. This is the gap closure and the substance of the issue. It replaces nothing: the existing tree scan stays as the baseline, and the range scan is added alongside it.

2. Full-history scan - opt-in, off by default, and not per-build. A separate job on a schedule plus manual dispatch, following the trigger pattern .github/workflows/update-dependencies.yml already uses (schedule plus workflow_dispatch) and the nightly_db_schedule anchor pattern in .circleci/config.yml. This covers Case B.

3. One-time onboarding scan - documentation only. The command to run, and the instruction to rotate rather than rewrite history. No CI involvement.

How the opt-in works, with no new variables

No new variable is needed for any of this:

• The existing #;< CI_GITLEAKS fence in .github/workflows/audit.yml already controls whether a project receives secret scanning at all.
• The scheduled full-history scan is a separately fenced job, so a project that does not select it does not receive it. That is the opt-in.
• VORTEX_CI_GITLEAKS_IGNORE_FAILURE already governs failure tolerance and applies to both scans unchanged.

Implementation notes

• Both providers currently invoke gitleaks dir .. The range scan needs the scanner's git mode with an explicit commit range rather than directory mode.
• .github/workflows/audit.yml checks out with no fetch-depth, so the clone is shallow and the base commit is absent. A range scan needs enough history fetched to contain the range - for pull requests, depth derived from the pull request's commit count is the documented idiom and is far cheaper than fetching everything.
• Edge cases that need explicit handling, or the step will fail confusingly rather than finding anything: the before-commit is all zeros on a branch's first push, and a force-push makes the range non-linear. Both need a defined fallback, such as scanning against the merge base or falling back to the tree scan.
• CircleCI clones with history available, so the range is easier to construct there, but it exposes no native base-commit variable for a pull request. The two providers will need slightly different range derivation - worth deciding rather than discovering.
• The scanner image pin is matched by a docker run custom manager in renovate.json, so a new job using the same invocation style stays version-tracked. A different invocation style would not be.

Acceptance criteria

• Every build scans the pushed commit range in addition to the working tree, so a secret that was committed and later removed is detected.
• Range scanning is enabled by default, since it is the routine gap this issue exists to close.
• Only enough history is fetched to cover the range - the routine build does not fetch the full repository.
• First-push and force-push cases have a defined fallback and do not fail the build spuriously.
• Both CI providers derive the range correctly for both pushes and pull requests.
• A full-history scan is available as a separate, opt-in, scheduled job with manual dispatch, and is not part of the per-build path.
• The opt-in is an installer fence, not a new runtime variable, and no new variable is introduced by this change.
• VORTEX_CI_GITLEAKS_IGNORE_FAILURE continues to govern failure tolerance for both scans.
• The existing allowlist file continues to suppress known false positives in all scan modes.
• Documentation covers the one-time onboarding scan as a command to run once, states plainly that the remediation for an exposed credential is rotation rather than history rewriting, and explains why the scheduled scan exists so it is not mistaken for a duplicate of the per-build one.
• Installer fixtures and snapshots are updated for the new step and the new fenced job.