diff --git a/README.md b/README.md index d0167bd..d87b08e 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![CI](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml/badge.svg)](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml) [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/) [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) -[![Tests: 501](https://img.shields.io/badge/tests-501%20passing-brightgreen.svg)](#testing) +[![Tests: 503](https://img.shields.io/badge/tests-503%20passing-brightgreen.svg)](#testing) **Financial anomaly detection and audit analytics over a 30,000-voucher general ledger.** @@ -58,7 +58,7 @@ Three design commitments drive everything else: | Anomalies caught by neither | **15** | | Isolation Forest | ROC-AUC **0.816**, precision 0.291, recall 0.286 | | Benford first-digit MAD | **0.00218** — close conformity | -| Test suite | **501 tests**, including dashboard render, warehouse and review-plan checks | +| Test suite | **503 tests**, including dashboard render, warehouse and review-plan checks | ### The most important number here is 98.2%, and it is a warning @@ -416,7 +416,13 @@ apostrophe in the CSV; the source values and selection logic are unchanged. The random controls have an inclusion probability of **60 / 29,888 = 0.2007%** within that remaining frame. The [selection record](outputs/reports/review_plan_summary.json) states the seed, policy, covered rules, population fingerprint and a SHA-256 of -the CSV. Those probabilities and weights apply **only** to the random route; +the CSV. The `selection-v2` source fingerprint retains 17 significant digits +for binary64 scores and amounts, so even a one-ULP change that reverses priority +cannot disappear into display rounding. Every field is quoted, including CR/LF +inside text, and the format is recorded separately from the review CSV checksum. +Earlier records use the original fingerprint policy; keep them with their +original CSV and selection record instead of replacing their digests. +Those probabilities and weights apply **only** to the random route; targeted cases do not support a population-wide exception estimate. A reviewer must examine documents before estimating anything. @@ -565,7 +571,7 @@ AuditLens/ │ ├── components.py # KPI cards, risk badges, charts, tables │ └── views/ # the six pages ├── sql/audit_queries.sql # 15 named business queries -├── tests/ # 501 tests, incl. cross-process reproducibility +├── tests/ # 503 tests, incl. cross-process reproducibility ├── docs/ │ ├── architecture.md # design decisions, data contracts, what is deliberately excluded │ ├── methodology.md # every threshold, every weight, every mistake @@ -579,7 +585,7 @@ AuditLens/ │ ├── notebook_content.py # what the three notebooks contain, kept apart from the builder │ ├── capture_screenshots.py # headless captures of all six dashboard pages, via DevTools Protocol │ └── ci_summary.py # headline figures for the CI job summary -├── .github/workflows/ci.yml # pipeline + 501 tests; Ubuntu 3.11/3.12 and Windows 3.12 +├── .github/workflows/ci.yml # pipeline + 503 tests; Ubuntu 3.11/3.12 and Windows 3.12 └── Makefile ``` @@ -752,7 +758,7 @@ builder, so two consecutive builds are byte-identical. ```bash pip install -r requirements.txt python src/run_pipeline.py # ~30 seconds, deterministic -python -m pytest tests/ -q # 501 tests +python -m pytest tests/ -q # 503 tests ``` Two consecutive runs produce byte-identical reports under `outputs/reports/`. This is @@ -760,7 +766,7 @@ enforced by `tests/test_reproducibility.py`, not assumed. ## Testing -501 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite. +503 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite. | File | Tests | What it pins down | |---|---|---| @@ -780,7 +786,7 @@ enforced by `tests/test_reproducibility.py`, not assumed. | `test_screenshots.py` | 11 | The six dashboard pages are named in four places — the `st.Page` titles, the `page_header` strings the views render, the capture tool's click labels and expected headings, and the committed PNGs — and all four must agree. Checked in both directions, so a rename cannot leave an orphan capture that the README still displays | | `test_readme_claims.py` | 27 | Every headline figure, the review-plan figures, all nine rules' flagged/precision/recall, the Benford table and every MAD quoted in the prose, the risk-band counts and values, the component weights, and the structural counts the prose quotes — checked against `outputs/reports/` and against the code | | `test_warehouse_contract.py` | 11 | Rejects partial SQLite builds, duplicate keys, orphaned or missing alerts and stale summary totals; preserves the prior warehouse when reconciliation fails and closes check/read connections before replacement | -| `test_review_plan.py` | 18 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, formula-safe CSV export, workpaper checksum and separate synthetic benchmark | +| `test_review_plan.py` | 20 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, one-ULP priority fingerprints, formula-safe CSV export, workpaper checksum and separate synthetic benchmark | | `test_review_outcomes.py` | 9 | A completed workpaper can edit review fields only; every finding needs evidence, counts and digests use the same CSV snapshot, and results remain route-specific counts without synthetic labels or population inference | Six of these are regression guards for bugs that were actually shipped during diff --git a/docs/methodology.md b/docs/methodology.md index c0c6b0c..7276d7f 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -516,7 +516,15 @@ from source documents. The targeted route has no design-based sampling weight. `review_plan_summary.json` records the policy, rule coverage, selection-input fingerprint (including the rule labels printed in selection reasons) and CSV -checksum. Missing or conflicting rule identities stop workpaper generation. +checksum. The source fingerprint now records format `selection-v2`: it sorts +the decision inputs, names the ledger/rule sections, quotes every CSV field and +retains 17 significant digits for binary64 numeric inputs. The earlier 12-digit +format could hash two distinct scores or amounts identically even when their +priority order changed. Review CSV precision remains a display choice; its own +`queue_sha256` binds the exact exported bytes. Existing original CSV/selection +record pairs remain usable for outcome capture; regenerating a source fingerprint +uses the new format and does not overwrite or authenticate an old record. +Missing or conflicting rule identities stop workpaper generation. `review_plan_benchmark.json` is a separate **synthetic-only** diagnostic computed after selection: 107 injected anomalies are in the workpaper, compared with 140 under pure risk ranking at the same diff --git a/outputs/reports/review_plan_summary.json b/outputs/reports/review_plan_summary.json index 30ef485..7b86cc5 100644 --- a/outputs/reports/review_plan_summary.json +++ b/outputs/reports/review_plan_summary.json @@ -36,7 +36,8 @@ "uncovered_rule_keys": [], "selected_amount_cny": 264116893.2, "selected_high_or_critical": 239, - "source_sha256": "17354261db2a984ad10590fbb953fc6a8ebc7ab1c8029ff4c839171441ce3f10", + "source_fingerprint_format": "selection-v2", + "source_sha256": "9ac4afed12dd05a2b4cdbe8dac2a8169a37ca7d963d636f09280f3f6d673d9f3", "interpretation": "Targeted cases are judgmental selections. The random probability and weight apply only to the non-targeted frame; no exception rate is inferred before review.", "queue_sha256": "d4898dfc31bfc3c03bd0175c8b23181d82433cc1ddcaf4161b7903740e92beec" } \ No newline at end of file diff --git a/src/review_plan.py b/src/review_plan.py index 934ca46..8319d28 100644 --- a/src/review_plan.py +++ b/src/review_plan.py @@ -13,6 +13,7 @@ from __future__ import annotations import argparse +import csv import hashlib import sys from dataclasses import dataclass @@ -47,6 +48,7 @@ "audit_risk_score", "risk_level", "rule_alert_count", "risk_reason_text", ) ROUTES = ("rule_coverage", "risk_priority", "random_control") +SOURCE_FINGERPRINT_FORMAT = "selection-v2" @dataclass(frozen=True) @@ -116,9 +118,15 @@ def selection_fingerprint(transactions: pd.DataFrame, alerts: pd.DataFrame) -> s if "rule_label" in alerts: rule_columns.append("rule_label") rules = alerts[rule_columns].sort_values(["transaction_id", "rule_key"]) - digest = hashlib.sha256() - for frame in (ledger, rules): - digest.update(frame.to_csv(index=False, lineterminator="\n", float_format="%.12g").encode()) + digest = hashlib.sha256(f"AuditLens {SOURCE_FINGERPRINT_FORMAT}\n".encode()) + for name, frame in (("ledger", ledger), ("rules", rules)): + digest.update(f"{name}\n".encode()) + # Selection uses full binary64 scores/amounts. Display precision can + # merge adjacent floats that reverse a priority tie. Seventeen digits + # retain binary64 values, and quoting all fields also preserves CR/LF. + digest.update(frame.to_csv( + index=False, lineterminator="\n", float_format="%.17g", quoting=csv.QUOTE_ALL, + ).encode()) return digest.hexdigest() @@ -232,6 +240,7 @@ def build_review_plan( "uncovered_rule_keys": sorted(population_rules - set(covered)), "selected_amount_cny": round(float(queue["debit_amount"].sum()), 2), "selected_high_or_critical": int(queue["risk_level"].isin(["High", "Critical"]).sum()), + "source_fingerprint_format": SOURCE_FINGERPRINT_FORMAT, "source_sha256": selection_fingerprint(transactions, alerts), "interpretation": ( "Targeted cases are judgmental selections. The random probability and weight " diff --git a/tests/test_review_plan.py b/tests/test_review_plan.py index 0c167a7..0aa5552 100644 --- a/tests/test_review_plan.py +++ b/tests/test_review_plan.py @@ -4,6 +4,7 @@ import hashlib +import numpy as np import pandas as pd import pytest @@ -85,6 +86,28 @@ def test_rule_labels_used_in_review_reasons_are_bound_to_source_fingerprint(popu assert "Control A" in revised.queue.loc[0, "selection_reason"] +@pytest.mark.parametrize("column,base", [("audit_risk_score", 80.0), ("debit_amount", 100.0)]) +def test_one_ulp_priority_change_cannot_keep_the_same_source_fingerprint(population, column, base) -> None: + transactions = population[0].iloc[:2].copy() + transactions["rule_alert_count"] = 0 + transactions["audit_risk_score"] = 80.0 + transactions["debit_amount"] = 100.0 + transactions[column] = [base, np.nextafter(base, np.inf)] + alerts = pd.DataFrame({ + "transaction_id": pd.Series(dtype=str), "rule_key": pd.Series(dtype=str), + }) + policy = ReviewPolicy(1, 0, 42) + original = build_review_plan(transactions, alerts, policy) + revised = transactions.copy() + revised[column] = transactions[column].iloc[::-1].to_numpy() + changed = build_review_plan(revised, alerts, policy) + + assert original.queue["transaction_id"].tolist() == ["TX2"] + assert changed.queue["transaction_id"].tolist() == ["TX1"] + assert original.summary["source_sha256"] != changed.summary["source_sha256"] + assert original.summary["source_fingerprint_format"] == "selection-v2" + + @pytest.mark.parametrize("corruption", ["blank_rule", "blank_label", "conflicting_label"]) def test_incomplete_rule_identity_cannot_enter_workpaper(population, corruption) -> None: transactions, alerts = population