Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![CI](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml/badge.svg)](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Tests: 501](https://img.shields.io/badge/tests-501%20passing-brightgreen.svg)](#testing)
[![Tests: 503](https://img.shields.io/badge/tests-503%20passing-brightgreen.svg)](#testing)

**Financial anomaly detection and audit analytics over a 30,000-voucher general ledger.**

Expand Down Expand Up @@ -58,7 +58,7 @@ Three design commitments drive everything else:
| Anomalies caught by neither | **15** |
| Isolation Forest | ROC-AUC **0.816**, precision 0.291, recall 0.286 |
| Benford first-digit MAD | **0.00218** — close conformity |
| Test suite | **501 tests**, including dashboard render, warehouse and review-plan checks |
| Test suite | **503 tests**, including dashboard render, warehouse and review-plan checks |

### The most important number here is 98.2%, and it is a warning

Expand Down Expand Up @@ -416,7 +416,13 @@ apostrophe in the CSV; the source values and selection logic are unchanged.
The random controls have an inclusion probability of **60 / 29,888 = 0.2007%**
within that remaining frame. The [selection record](outputs/reports/review_plan_summary.json)
states the seed, policy, covered rules, population fingerprint and a SHA-256 of
the CSV. Those probabilities and weights apply **only** to the random route;
the CSV. The `selection-v2` source fingerprint retains 17 significant digits
for binary64 scores and amounts, so even a one-ULP change that reverses priority
cannot disappear into display rounding. Every field is quoted, including CR/LF
inside text, and the format is recorded separately from the review CSV checksum.
Earlier records use the original fingerprint policy; keep them with their
original CSV and selection record instead of replacing their digests.
Those probabilities and weights apply **only** to the random route;
targeted cases do not support a population-wide exception estimate. A reviewer
must examine documents before estimating anything.

Expand Down Expand Up @@ -565,7 +571,7 @@ AuditLens/
│ ├── components.py # KPI cards, risk badges, charts, tables
│ └── views/ # the six pages
├── sql/audit_queries.sql # 15 named business queries
├── tests/ # 501 tests, incl. cross-process reproducibility
├── tests/ # 503 tests, incl. cross-process reproducibility
├── docs/
│ ├── architecture.md # design decisions, data contracts, what is deliberately excluded
│ ├── methodology.md # every threshold, every weight, every mistake
Expand All @@ -579,7 +585,7 @@ AuditLens/
│ ├── notebook_content.py # what the three notebooks contain, kept apart from the builder
│ ├── capture_screenshots.py # headless captures of all six dashboard pages, via DevTools Protocol
│ └── ci_summary.py # headline figures for the CI job summary
├── .github/workflows/ci.yml # pipeline + 501 tests; Ubuntu 3.11/3.12 and Windows 3.12
├── .github/workflows/ci.yml # pipeline + 503 tests; Ubuntu 3.11/3.12 and Windows 3.12
└── Makefile
```

Expand Down Expand Up @@ -752,15 +758,15 @@ builder, so two consecutive builds are byte-identical.
```bash
pip install -r requirements.txt
python src/run_pipeline.py # ~30 seconds, deterministic
python -m pytest tests/ -q # 501 tests
python -m pytest tests/ -q # 503 tests
```

Two consecutive runs produce byte-identical reports under `outputs/reports/`. This is
enforced by `tests/test_reproducibility.py`, not assumed.

## Testing

501 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite.
503 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite.

| File | Tests | What it pins down |
|---|---|---|
Expand All @@ -780,7 +786,7 @@ enforced by `tests/test_reproducibility.py`, not assumed.
| `test_screenshots.py` | 11 | The six dashboard pages are named in four places — the `st.Page` titles, the `page_header` strings the views render, the capture tool's click labels and expected headings, and the committed PNGs — and all four must agree. Checked in both directions, so a rename cannot leave an orphan capture that the README still displays |
| `test_readme_claims.py` | 27 | Every headline figure, the review-plan figures, all nine rules' flagged/precision/recall, the Benford table and every MAD quoted in the prose, the risk-band counts and values, the component weights, and the structural counts the prose quotes — checked against `outputs/reports/` and against the code |
| `test_warehouse_contract.py` | 11 | Rejects partial SQLite builds, duplicate keys, orphaned or missing alerts and stale summary totals; preserves the prior warehouse when reconciliation fails and closes check/read connections before replacement |
| `test_review_plan.py` | 18 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, formula-safe CSV export, workpaper checksum and separate synthetic benchmark |
| `test_review_plan.py` | 20 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, one-ULP priority fingerprints, formula-safe CSV export, workpaper checksum and separate synthetic benchmark |
| `test_review_outcomes.py` | 9 | A completed workpaper can edit review fields only; every finding needs evidence, counts and digests use the same CSV snapshot, and results remain route-specific counts without synthetic labels or population inference |

Six of these are regression guards for bugs that were actually shipped during
Expand Down
10 changes: 9 additions & 1 deletion docs/methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -516,7 +516,15 @@ from source documents. The targeted route has no design-based sampling weight.

`review_plan_summary.json` records the policy, rule coverage, selection-input
fingerprint (including the rule labels printed in selection reasons) and CSV
checksum. Missing or conflicting rule identities stop workpaper generation.
checksum. The source fingerprint now records format `selection-v2`: it sorts
the decision inputs, names the ledger/rule sections, quotes every CSV field and
retains 17 significant digits for binary64 numeric inputs. The earlier 12-digit
format could hash two distinct scores or amounts identically even when their
priority order changed. Review CSV precision remains a display choice; its own
`queue_sha256` binds the exact exported bytes. Existing original CSV/selection
record pairs remain usable for outcome capture; regenerating a source fingerprint
uses the new format and does not overwrite or authenticate an old record.
Missing or conflicting rule identities stop workpaper generation.
`review_plan_benchmark.json` is a separate
**synthetic-only** diagnostic computed after selection: 107 injected anomalies
are in the workpaper, compared with 140 under pure risk ranking at the same
Expand Down
3 changes: 2 additions & 1 deletion outputs/reports/review_plan_summary.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@
"uncovered_rule_keys": [],
"selected_amount_cny": 264116893.2,
"selected_high_or_critical": 239,
"source_sha256": "17354261db2a984ad10590fbb953fc6a8ebc7ab1c8029ff4c839171441ce3f10",
"source_fingerprint_format": "selection-v2",
"source_sha256": "9ac4afed12dd05a2b4cdbe8dac2a8169a37ca7d963d636f09280f3f6d673d9f3",
"interpretation": "Targeted cases are judgmental selections. The random probability and weight apply only to the non-targeted frame; no exception rate is inferred before review.",
"queue_sha256": "d4898dfc31bfc3c03bd0175c8b23181d82433cc1ddcaf4161b7903740e92beec"
}
15 changes: 12 additions & 3 deletions src/review_plan.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from __future__ import annotations

import argparse
import csv
import hashlib
import sys
from dataclasses import dataclass
Expand Down Expand Up @@ -47,6 +48,7 @@
"audit_risk_score", "risk_level", "rule_alert_count", "risk_reason_text",
)
ROUTES = ("rule_coverage", "risk_priority", "random_control")
SOURCE_FINGERPRINT_FORMAT = "selection-v2"


@dataclass(frozen=True)
Expand Down Expand Up @@ -116,9 +118,15 @@ def selection_fingerprint(transactions: pd.DataFrame, alerts: pd.DataFrame) -> s
if "rule_label" in alerts:
rule_columns.append("rule_label")
rules = alerts[rule_columns].sort_values(["transaction_id", "rule_key"])
digest = hashlib.sha256()
for frame in (ledger, rules):
digest.update(frame.to_csv(index=False, lineterminator="\n", float_format="%.12g").encode())
digest = hashlib.sha256(f"AuditLens {SOURCE_FINGERPRINT_FORMAT}\n".encode())
for name, frame in (("ledger", ledger), ("rules", rules)):
digest.update(f"{name}\n".encode())
# Selection uses full binary64 scores/amounts. Display precision can
# merge adjacent floats that reverse a priority tie. Seventeen digits
# retain binary64 values, and quoting all fields also preserves CR/LF.
digest.update(frame.to_csv(
index=False, lineterminator="\n", float_format="%.17g", quoting=csv.QUOTE_ALL,
).encode())
return digest.hexdigest()


Expand Down Expand Up @@ -232,6 +240,7 @@ def build_review_plan(
"uncovered_rule_keys": sorted(population_rules - set(covered)),
"selected_amount_cny": round(float(queue["debit_amount"].sum()), 2),
"selected_high_or_critical": int(queue["risk_level"].isin(["High", "Critical"]).sum()),
"source_fingerprint_format": SOURCE_FINGERPRINT_FORMAT,
"source_sha256": selection_fingerprint(transactions, alerts),
"interpretation": (
"Targeted cases are judgmental selections. The random probability and weight "
Expand Down
23 changes: 23 additions & 0 deletions tests/test_review_plan.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

import hashlib

import numpy as np
import pandas as pd
import pytest

Expand Down Expand Up @@ -85,6 +86,28 @@ def test_rule_labels_used_in_review_reasons_are_bound_to_source_fingerprint(popu
assert "Control A" in revised.queue.loc[0, "selection_reason"]


@pytest.mark.parametrize("column,base", [("audit_risk_score", 80.0), ("debit_amount", 100.0)])
def test_one_ulp_priority_change_cannot_keep_the_same_source_fingerprint(population, column, base) -> None:
transactions = population[0].iloc[:2].copy()
transactions["rule_alert_count"] = 0
transactions["audit_risk_score"] = 80.0
transactions["debit_amount"] = 100.0
transactions[column] = [base, np.nextafter(base, np.inf)]
alerts = pd.DataFrame({
"transaction_id": pd.Series(dtype=str), "rule_key": pd.Series(dtype=str),
})
policy = ReviewPolicy(1, 0, 42)
original = build_review_plan(transactions, alerts, policy)
revised = transactions.copy()
revised[column] = transactions[column].iloc[::-1].to_numpy()
changed = build_review_plan(revised, alerts, policy)

assert original.queue["transaction_id"].tolist() == ["TX2"]
assert changed.queue["transaction_id"].tolist() == ["TX1"]
assert original.summary["source_sha256"] != changed.summary["source_sha256"]
assert original.summary["source_fingerprint_format"] == "selection-v2"


@pytest.mark.parametrize("corruption", ["blank_rule", "blank_label", "conflicting_label"])
def test_incomplete_rule_identity_cannot_enter_workpaper(population, corruption) -> None:
transactions, alerts = population
Expand Down
Loading