diff --git a/README.md b/README.md index d87b08e..ff333c2 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![CI](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml/badge.svg)](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml) [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/) [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE) -[![Tests: 503](https://img.shields.io/badge/tests-503%20passing-brightgreen.svg)](#testing) +[![Tests: 508](https://img.shields.io/badge/tests-508%20passing-brightgreen.svg)](#testing) **Financial anomaly detection and audit analytics over a 30,000-voucher general ledger.** @@ -58,7 +58,7 @@ Three design commitments drive everything else: | Anomalies caught by neither | **15** | | Isolation Forest | ROC-AUC **0.816**, precision 0.291, recall 0.286 | | Benford first-digit MAD | **0.00218** — close conformity | -| Test suite | **503 tests**, including dashboard render, warehouse and review-plan checks | +| Test suite | **508 tests**, including dashboard render, warehouse and review-plan checks | ### The most important number here is 98.2%, and it is a warning @@ -410,8 +410,15 @@ seeded random controls from the **29,888 vouchers left after targeted selection* It includes each case's selection reason, triggered rules, risk explanation, and blank outcome, evidence-reference and reviewer-note fields. No injected answer-key column enters the queue or its selection fingerprint. -Text cells that could be read as spreadsheet formulas are prefixed with an -apostrophe in the CSV; the source values and selection logic are unchanged. +Review CSV values and headers that could be read as spreadsheet formulas, +including fullwidth variants, receive a visible `text: ` prefix. Leading +control/BOM characters are marked, and an existing `text: ` prefix is escaped +again to keep distinct values distinct. Embedded CR, LF and CRLF stay quoted +inside cells; record endings remain LF. Source values and selection logic are +unchanged. This policy covers the review queue and high-risk extract; internal +data files retain their raw values. CSV remains a review copy, with behavior +depending on spreadsheet import/editing settings; see +[OWASP's guidance](https://owasp.org/www-community/attacks/CSV_Injection). The random controls have an inclusion probability of **60 / 29,888 = 0.2007%** within that remaining frame. The [selection record](outputs/reports/review_plan_summary.json) @@ -571,7 +578,7 @@ AuditLens/ │ ├── components.py # KPI cards, risk badges, charts, tables │ └── views/ # the six pages ├── sql/audit_queries.sql # 15 named business queries -├── tests/ # 503 tests, incl. cross-process reproducibility +├── tests/ # 508 tests, incl. cross-process reproducibility ├── docs/ │ ├── architecture.md # design decisions, data contracts, what is deliberately excluded │ ├── methodology.md # every threshold, every weight, every mistake @@ -585,7 +592,7 @@ AuditLens/ │ ├── notebook_content.py # what the three notebooks contain, kept apart from the builder │ ├── capture_screenshots.py # headless captures of all six dashboard pages, via DevTools Protocol │ └── ci_summary.py # headline figures for the CI job summary -├── .github/workflows/ci.yml # pipeline + 503 tests; Ubuntu 3.11/3.12 and Windows 3.12 +├── .github/workflows/ci.yml # pipeline + 508 tests; Ubuntu 3.11/3.12 and Windows 3.12 └── Makefile ``` @@ -758,7 +765,7 @@ builder, so two consecutive builds are byte-identical. ```bash pip install -r requirements.txt python src/run_pipeline.py # ~30 seconds, deterministic -python -m pytest tests/ -q # 503 tests +python -m pytest tests/ -q # 508 tests ``` Two consecutive runs produce byte-identical reports under `outputs/reports/`. This is @@ -766,7 +773,7 @@ enforced by `tests/test_reproducibility.py`, not assumed. ## Testing -503 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite. +508 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite. | File | Tests | What it pins down | |---|---|---| @@ -776,7 +783,7 @@ enforced by `tests/test_reproducibility.py`, not assumed. | `test_risk_scoring.py` | 55 | Component scores, the noisy-OR combination, band boundaries, `risk_reasons` wording | | `test_dashboard.py` | 65 | Renders all six pages in-process via `AppTest`, checks the workpaper handoff, control catalogue and distinct-voucher share, and pins the no-ground-truth guard at both the rendered-output and grid-builder level | | `test_feature_engineering.py` | 9 | The model's feature matrix: no ground-truth column can reach it, the matrix and its descriptions cover exactly the same set, and the committed `model_metrics.json` records the features the code actually built | -| `test_utils.py` | 19 | `as_flag_series` across every dtype the label takes, including the two string traps | +| `test_utils.py` | 23 | `as_flag_series` across every dtype the label takes; review CSV formulas, multiline cells, numbers, missing values, prefix collisions and empty headers | | `test_ci_summary.py` | 9 | The CI job summary renders, carries its benchmark caveat, and degrades to a dash on schema drift rather than breaking the build | | `test_reporting.py` | 8 | The flag-vs-anomaly distinction in `detector_overlap`, stable ordering for equal risk scores, and formula-safe text in the review extract | | `test_reproducibility.py` | 4 | Runs the generator in two subprocesses with different `PYTHONHASHSEED` values and compares hashes | @@ -786,7 +793,7 @@ enforced by `tests/test_reproducibility.py`, not assumed. | `test_screenshots.py` | 11 | The six dashboard pages are named in four places — the `st.Page` titles, the `page_header` strings the views render, the capture tool's click labels and expected headings, and the committed PNGs — and all four must agree. Checked in both directions, so a rename cannot leave an orphan capture that the README still displays | | `test_readme_claims.py` | 27 | Every headline figure, the review-plan figures, all nine rules' flagged/precision/recall, the Benford table and every MAD quoted in the prose, the risk-band counts and values, the component weights, and the structural counts the prose quotes — checked against `outputs/reports/` and against the code | | `test_warehouse_contract.py` | 11 | Rejects partial SQLite builds, duplicate keys, orphaned or missing alerts and stale summary totals; preserves the prior warehouse when reconciliation fails and closes check/read connections before replacement | -| `test_review_plan.py` | 20 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, one-ULP priority fingerprints, formula-safe CSV export, workpaper checksum and separate synthetic benchmark | +| `test_review_plan.py` | 21 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, one-ULP priority fingerprints, formula-safe/multiline CSV handoff, workpaper checksum and separate synthetic benchmark | | `test_review_outcomes.py` | 9 | A completed workpaper can edit review fields only; every finding needs evidence, counts and digests use the same CSV snapshot, and results remain route-specific counts without synthetic labels or population inference | Six of these are regression guards for bugs that were actually shipped during diff --git a/docs/methodology.md b/docs/methodology.md index 7276d7f..8f2f05a 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -525,6 +525,13 @@ priority order changed. Review CSV precision remains a display choice; its own record pairs remain usable for outcome capture; regenerating a source fingerprint uses the new format and does not overwrite or authenticate an old record. Missing or conflicting rule identities stop workpaper generation. +Auditor-facing review queues and high-risk extracts mark formula-shaped text +(ASCII/fullwidth prefixes and leading control/BOM characters) with visible +`text: ` prefixes. A pre-existing marker is escaped again to prevent collisions. +CR/LF/CRLF within a value is quoted without changing it; records retain LF. +Only these review copies are transformed; raw data tables and selection inputs +remain unchanged. Existing original CSV/selection-record pairs still validate +under their retained bytes; re-exporting uses the current CSV policy. `review_plan_benchmark.json` is a separate **synthetic-only** diagnostic computed after selection: 107 injected anomalies are in the workpaper, compared with 140 under pure risk ranking at the same diff --git a/src/reporting.py b/src/reporting.py index 64f26dc..2f80d26 100644 --- a/src/reporting.py +++ b/src/reporting.py @@ -54,8 +54,8 @@ get_logger, load_dataframe, load_json, + review_csv_text, save_json, - spreadsheet_safe_cell, ) LOGGER = get_logger(__name__) @@ -431,7 +431,7 @@ def write_high_risk_extract(df: pd.DataFrame) -> Path: extract = df.loc[df["risk_level"].isin(HIGH_RISK_LEVELS), columns].sort_values( ["audit_risk_score", "transaction_id"], ascending=[False, True] ) - extract.map(spreadsheet_safe_cell).to_csv(HIGH_RISK_CSV, index=False, encoding="utf-8-sig") + HIGH_RISK_CSV.write_bytes(review_csv_text(extract).encode("utf-8-sig")) LOGGER.info("Wrote %s (%s vouchers)", HIGH_RISK_CSV.name, len(extract)) return HIGH_RISK_CSV diff --git a/src/review_plan.py b/src/review_plan.py index 8319d28..d8932ab 100644 --- a/src/review_plan.py +++ b/src/review_plan.py @@ -37,8 +37,8 @@ ensure_directories, get_logger, load_dataframe, + review_csv_text, save_json, - spreadsheet_safe_cell, ) LOGGER = get_logger(__name__) @@ -258,8 +258,7 @@ def write_review_plan( """Write the auditor queue and a checksum-bound selection record.""" queue_path.parent.mkdir(parents=True, exist_ok=True) summary_path.parent.mkdir(parents=True, exist_ok=True) - export_queue = plan.queue.map(spreadsheet_safe_cell) - contents = export_queue.to_csv(index=False, lineterminator="\n", float_format="%.10g") + contents = review_csv_text(plan.queue, float_format="%.10g") summary = {**plan.summary, "queue_sha256": hashlib.sha256(contents.encode()).hexdigest()} queue_path.write_bytes(contents.encode("utf-8")) save_json(summary, summary_path) diff --git a/src/utils.py b/src/utils.py index 92cf8db..40c5905 100644 --- a/src/utils.py +++ b/src/utils.py @@ -422,12 +422,29 @@ def load_json(path: Path) -> dict[str, Any]: def spreadsheet_safe_cell(value: Any) -> Any: - """Prefix formula-like text in auditor-facing CSV exports.""" - if isinstance(value, str) and value.lstrip().startswith(("=", "+", "-", "@")): - return "'" + value + """Keep untrusted review text visibly textual; source data stays unchanged.""" + if isinstance(value, str) and value: + stripped = value.lstrip() + if (value.startswith("text: ") or ord(value[0]) < 32 or ord(value[0]) == 127 + or value[0] == "\ufeff" + or stripped.startswith(("=", "+", "-", "@", "＝", "＋", "－", "＠"))): + return "text: " + value return value +def review_csv_text(frame: pd.DataFrame, *, float_format: str | None = None) -> str: + """Write review CSV with LF records and quoted embedded CR/LF unchanged.""" + exported = frame.map(spreadsheet_safe_cell).rename(columns=spreadsheet_safe_cell) + options = {"index": False, "lineterminator": "\r\n", "float_format": float_format} + # Using CRLF for each record makes pandas quote both CR and LF inside a + # field. Normalize only that record's final terminator, never its cell data. + header = exported.iloc[:0].to_csv(**options).removesuffix("\r\n") + "\n" + return header + "".join( + exported.iloc[index:index + 1].to_csv(header=False, **options).removesuffix("\r\n") + "\n" + for index in range(len(exported)) + ) + + def save_dataframe(df: pd.DataFrame, path: Path) -> Path: """Persist a dataframe to parquet (or CSV when the suffix is ``.csv``).""" path.parent.mkdir(parents=True, exist_ok=True) diff --git a/tests/test_reporting.py b/tests/test_reporting.py index c48ca6a..7ded347 100644 --- a/tests/test_reporting.py +++ b/tests/test_reporting.py @@ -63,10 +63,10 @@ def test_high_risk_extract_escapes_formula_like_text_without_changing_source( reporting.write_high_risk_extract(vouchers) row = pd.read_csv(output, dtype=str, keep_default_na=False).iloc[0] - assert row["transaction_id"] == "'" + voucher_id - assert row["vendor_name"] == "' +SUM(1,2)" - assert row["description"] == "'-1+2" - assert row["risk_reason_text"] == "'@SUM(1,2)" + assert row["transaction_id"] == "text: " + voucher_id + assert row["vendor_name"] == "text: +SUM(1,2)" + assert row["description"] == "text: -1+2" + assert row["risk_reason_text"] == "text: @SUM(1,2)" assert row["audit_risk_score"] == "82.0" assert vouchers.loc[0, "transaction_id"] == voucher_id assert vouchers.loc[0, "vendor_name"] == " +SUM(1,2)" diff --git a/tests/test_review_plan.py b/tests/test_review_plan.py index 0aa5552..19048dd 100644 --- a/tests/test_review_plan.py +++ b/tests/test_review_plan.py @@ -3,6 +3,8 @@ from __future__ import annotations import hashlib +import csv +import io import numpy as np import pandas as pd @@ -172,10 +174,10 @@ def test_workpaper_escapes_spreadsheet_formulas_without_changing_selection(popul csv_path, json_path = tmp_path / "selection.csv", tmp_path / "selection.json" write_review_plan(plan, csv_path, json_path) exported = pd.read_csv(csv_path, dtype=str, keep_default_na=False) - selected = exported.loc[exported["transaction_id"] == "'" + dangerous_id].iloc[0] - assert selected["vendor_id"] == "' +SUM(1,2)" - assert selected["risk_reason_text"] == "'-1+2" - assert selected["risk_level"] == "'@SUM(1,2)" + selected = exported.loc[exported["transaction_id"] == "text: " + dangerous_id].iloc[0] + assert selected["vendor_id"] == "text: +SUM(1,2)" + assert selected["risk_reason_text"] == "text: -1+2" + assert selected["risk_level"] == "text: @SUM(1,2)" assert selected["selection_reason"].startswith("Covers procedures: ") assert plan.queue["transaction_id"].tolist() == original_ids assert plan.queue.loc[plan.queue["transaction_id"] == dangerous_id, "vendor_id"].iloc[0] == ( @@ -197,3 +199,20 @@ def test_synthetic_benchmark_is_computed_only_after_the_label_blind_selection(po changed["anomaly_label"] = "0" # parquet also stores labels as strings assert build_review_plan(changed, alerts, ReviewPolicy(5, 0.2, 42)).queue.equals(plan.queue) assert benchmark_review_plan(plan, changed)["plan_injected_anomalies_found"] == 0 + + +def test_multiline_fullwidth_id_survives_workpaper_and_outcome_capture(population, tmp_path) -> None: + transactions, alerts = (frame.copy() for frame in population) + identifier = "＝1\rsecond\nthird\r\nfourth" + transactions.loc[transactions["transaction_id"] == "TX2", "transaction_id"] = identifier + alerts.loc[alerts["transaction_id"] == "TX2", "transaction_id"] = identifier + plan = build_review_plan(transactions, alerts, ReviewPolicy(5, 0.2, 42)) + original, record, completed = (tmp_path / name for name in ("original.csv", "record.json", "copy.csv")) + write_review_plan(plan, original, record) + raw = original.read_bytes() + rows = list(csv.DictReader(io.StringIO(raw.decode(), newline=""))) + assert rows[0]["transaction_id"] == "text: " + identifier + assert len(rows) == 5 + assert plan.queue.loc[0, "transaction_id"] == identifier + completed.write_bytes(raw) + assert summarize_completed_review(completed, original, record)["selected_count"] == 5 diff --git a/tests/test_utils.py b/tests/test_utils.py index 8401c56..f7824b4 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -9,6 +9,8 @@ from __future__ import annotations +import csv +import io import sys from pathlib import Path @@ -20,7 +22,35 @@ if str(PROJECT_ROOT) not in sys.path: sys.path.insert(0, str(PROJECT_ROOT)) -from src.utils import REASON_SEPARATOR, as_flag_series, as_list, as_reason_list +from src.utils import REASON_SEPARATOR, as_flag_series, as_list, as_reason_list, review_csv_text + + +def test_review_csv_marks_formula_variants_and_control_prefixes_as_text() -> None: + values = ["=1", "+1", "-1", "@SUM(1)", "＝1", "＋1", "－1", "＠SUM(1)", + " ＝1", "\t=1", "\r=1", "\n=1", "\ufeff=1", "\x00=1", "\x7f=1"] + frame = pd.DataFrame({"note": values}) + rows = list(csv.reader(io.StringIO(review_csv_text(frame), newline=""))) + assert rows == [["note"], *[["text: " + value] for value in values]] + assert frame["note"].tolist() == values + + +def test_review_csv_preserves_multiline_cells_numbers_and_missing_values() -> None: + note = 'A\rB\nC\r\nD, "quoted"' + frame = pd.DataFrame({"note": [note, "plain"], "amount": [-12.5, np.nan]}) + rows = list(csv.reader(io.StringIO(review_csv_text(frame, float_format="%.10g"), newline=""))) + assert rows == [["note", "amount"], [note, "-12.5"], ["plain", ""]] + + +def test_review_csv_prefix_is_collision_safe() -> None: + values = ["=1", "text: =1", "text: text: =1"] + frame = pd.DataFrame({"note": values}) + rows = list(csv.reader(io.StringIO(review_csv_text(frame), newline=""))) + assert rows[1:] == [["text: " + value] for value in values] + + +def test_review_csv_empty_table_still_has_a_safe_header() -> None: + text = review_csv_text(pd.DataFrame(columns=["＝field"])) + assert list(csv.reader(io.StringIO(text, newline=""))) == [["text: ＝field"]] class TestAsFlagSeries: