Preserve review CSV identities and fullwidth formula protection

README.md

@@ -3,7 +3,7 @@
 [![CI](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml/badge.svg)](https://github.com/dev-belly/AuditLens/actions/workflows/ci.yml)
 [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
-[![Tests: 503](https://img.shields.io/badge/tests-503%20passing-brightgreen.svg)](#testing)
+[![Tests: 508](https://img.shields.io/badge/tests-508%20passing-brightgreen.svg)](#testing)
 
 **Financial anomaly detection and audit analytics over a 30,000-voucher general ledger.**
 
@@ -58,7 +58,7 @@ Three design commitments drive everything else:
 | Anomalies caught by neither | **15** |
 | Isolation Forest | ROC-AUC **0.816**, precision 0.291, recall 0.286 |
 | Benford first-digit MAD | **0.00218** — close conformity |
-| Test suite | **503 tests**, including dashboard render, warehouse and review-plan checks |
+| Test suite | **508 tests**, including dashboard render, warehouse and review-plan checks |
 
 ### The most important number here is 98.2%, and it is a warning
 
@@ -410,8 +410,15 @@ seeded random controls from the **29,888 vouchers left after targeted selection*
 It includes each case's selection reason, triggered rules, risk explanation, and
 blank outcome, evidence-reference and reviewer-note fields. No injected answer-key
 column enters the queue or its selection fingerprint.
-Text cells that could be read as spreadsheet formulas are prefixed with an
-apostrophe in the CSV; the source values and selection logic are unchanged.
+Review CSV values and headers that could be read as spreadsheet formulas,
+including fullwidth variants, receive a visible `text: ` prefix. Leading
+control/BOM characters are marked, and an existing `text: ` prefix is escaped
+again to keep distinct values distinct. Embedded CR, LF and CRLF stay quoted
+inside cells; record endings remain LF. Source values and selection logic are
+unchanged. This policy covers the review queue and high-risk extract; internal
+data files retain their raw values. CSV remains a review copy, with behavior
+depending on spreadsheet import/editing settings; see
+[OWASP's guidance](https://owasp.org/www-community/attacks/CSV_Injection).
 
 The random controls have an inclusion probability of **60 / 29,888 = 0.2007%**
 within that remaining frame. The [selection record](outputs/reports/review_plan_summary.json)
@@ -571,7 +578,7 @@ AuditLens/
 │   ├── components.py            # KPI cards, risk badges, charts, tables
 │   └── views/                   # the six pages
 ├── sql/audit_queries.sql        # 15 named business queries
-├── tests/                       # 503 tests, incl. cross-process reproducibility
+├── tests/                       # 508 tests, incl. cross-process reproducibility
 ├── docs/
 │   ├── architecture.md          # design decisions, data contracts, what is deliberately excluded
 │   ├── methodology.md           # every threshold, every weight, every mistake
@@ -585,7 +592,7 @@ AuditLens/
 │   ├── notebook_content.py      # what the three notebooks contain, kept apart from the builder
 │   ├── capture_screenshots.py   # headless captures of all six dashboard pages, via DevTools Protocol
 │   └── ci_summary.py            # headline figures for the CI job summary
-├── .github/workflows/ci.yml     # pipeline + 503 tests; Ubuntu 3.11/3.12 and Windows 3.12
+├── .github/workflows/ci.yml     # pipeline + 508 tests; Ubuntu 3.11/3.12 and Windows 3.12
 └── Makefile
 ```
 
@@ -758,15 +765,15 @@ builder, so two consecutive builds are byte-identical.
 ```bash
 pip install -r requirements.txt
 python src/run_pipeline.py     # ~30 seconds, deterministic
-python -m pytest tests/ -q     # 503 tests
+python -m pytest tests/ -q     # 508 tests
 ```
 
 Two consecutive runs produce byte-identical reports under `outputs/reports/`. This is
 enforced by `tests/test_reproducibility.py`, not assumed.
 
 ## Testing
 
-503 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite.
+508 tests. `make test` runs the lot; `make test-fast` skips the dashboard render suite.
 
 | File | Tests | What it pins down |
 |---|---|---|
@@ -776,7 +783,7 @@ enforced by `tests/test_reproducibility.py`, not assumed.
 | `test_risk_scoring.py` | 55 | Component scores, the noisy-OR combination, band boundaries, `risk_reasons` wording |
 | `test_dashboard.py` | 65 | Renders all six pages in-process via `AppTest`, checks the workpaper handoff, control catalogue and distinct-voucher share, and pins the no-ground-truth guard at both the rendered-output and grid-builder level |
 | `test_feature_engineering.py` | 9 | The model's feature matrix: no ground-truth column can reach it, the matrix and its descriptions cover exactly the same set, and the committed `model_metrics.json` records the features the code actually built |
-| `test_utils.py` | 19 | `as_flag_series` across every dtype the label takes, including the two string traps |
+| `test_utils.py` | 23 | `as_flag_series` across every dtype the label takes; review CSV formulas, multiline cells, numbers, missing values, prefix collisions and empty headers |
 | `test_ci_summary.py` | 9 | The CI job summary renders, carries its benchmark caveat, and degrades to a dash on schema drift rather than breaking the build |
 | `test_reporting.py` | 8 | The flag-vs-anomaly distinction in `detector_overlap`, stable ordering for equal risk scores, and formula-safe text in the review extract |
 | `test_reproducibility.py` | 4 | Runs the generator in two subprocesses with different `PYTHONHASHSEED` values and compares hashes |
@@ -786,7 +793,7 @@ enforced by `tests/test_reproducibility.py`, not assumed.
 | `test_screenshots.py` | 11 | The six dashboard pages are named in four places — the `st.Page` titles, the `page_header` strings the views render, the capture tool's click labels and expected headings, and the committed PNGs — and all four must agree. Checked in both directions, so a rename cannot leave an orphan capture that the README still displays |
 | `test_readme_claims.py` | 27 | Every headline figure, the review-plan figures, all nine rules' flagged/precision/recall, the Benford table and every MAD quoted in the prose, the risk-band counts and values, the component weights, and the structural counts the prose quotes — checked against `outputs/reports/` and against the code |
 | `test_warehouse_contract.py` | 11 | Rejects partial SQLite builds, duplicate keys, orphaned or missing alerts and stale summary totals; preserves the prior warehouse when reconciliation fails and closes check/read connections before replacement |
-| `test_review_plan.py` | 20 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, one-ULP priority fingerprints, formula-safe CSV export, workpaper checksum and separate synthetic benchmark |
+| `test_review_plan.py` | 21 | Fixed-budget coverage and risk ranking, random inclusion probabilities, label blindness, input-order invariance, complete rule identity, one-ULP priority fingerprints, formula-safe/multiline CSV handoff, workpaper checksum and separate synthetic benchmark |
 | `test_review_outcomes.py` | 9 | A completed workpaper can edit review fields only; every finding needs evidence, counts and digests use the same CSV snapshot, and results remain route-specific counts without synthetic labels or population inference |
 
 Six of these are regression guards for bugs that were actually shipped during

docs/methodology.md

@@ -525,6 +525,13 @@ priority order changed. Review CSV precision remains a display choice; its own
 record pairs remain usable for outcome capture; regenerating a source fingerprint
 uses the new format and does not overwrite or authenticate an old record.
 Missing or conflicting rule identities stop workpaper generation.
+Auditor-facing review queues and high-risk extracts mark formula-shaped text
+(ASCII/fullwidth prefixes and leading control/BOM characters) with visible
+`text: ` prefixes. A pre-existing marker is escaped again to prevent collisions.
+CR/LF/CRLF within a value is quoted without changing it; records retain LF.
+Only these review copies are transformed; raw data tables and selection inputs
+remain unchanged. Existing original CSV/selection-record pairs still validate
+under their retained bytes; re-exporting uses the current CSV policy.
 `review_plan_benchmark.json` is a separate
 **synthetic-only** diagnostic computed after selection: 107 injected anomalies
 are in the workpaper, compared with 140 under pure risk ranking at the same

src/reporting.py

@@ -54,8 +54,8 @@
     get_logger,
     load_dataframe,
     load_json,
+    review_csv_text,
     save_json,
-    spreadsheet_safe_cell,
 )
 
 LOGGER = get_logger(__name__)
@@ -431,7 +431,7 @@ def write_high_risk_extract(df: pd.DataFrame) -> Path:
     extract = df.loc[df["risk_level"].isin(HIGH_RISK_LEVELS), columns].sort_values(
         ["audit_risk_score", "transaction_id"], ascending=[False, True]
     )
-    extract.map(spreadsheet_safe_cell).to_csv(HIGH_RISK_CSV, index=False, encoding="utf-8-sig")
+    HIGH_RISK_CSV.write_bytes(review_csv_text(extract).encode("utf-8-sig"))
     LOGGER.info("Wrote %s (%s vouchers)", HIGH_RISK_CSV.name, len(extract))
     return HIGH_RISK_CSV
 

src/review_plan.py

@@ -37,8 +37,8 @@
     ensure_directories,
     get_logger,
     load_dataframe,
+    review_csv_text,
     save_json,
-    spreadsheet_safe_cell,
 )
 
 LOGGER = get_logger(__name__)
@@ -258,8 +258,7 @@ def write_review_plan(
     """Write the auditor queue and a checksum-bound selection record."""
     queue_path.parent.mkdir(parents=True, exist_ok=True)
     summary_path.parent.mkdir(parents=True, exist_ok=True)
-    export_queue = plan.queue.map(spreadsheet_safe_cell)
-    contents = export_queue.to_csv(index=False, lineterminator="\n", float_format="%.10g")
+    contents = review_csv_text(plan.queue, float_format="%.10g")
     summary = {**plan.summary, "queue_sha256": hashlib.sha256(contents.encode()).hexdigest()}
     queue_path.write_bytes(contents.encode("utf-8"))
     save_json(summary, summary_path)

src/utils.py

@@ -422,12 +422,29 @@ def load_json(path: Path) -> dict[str, Any]:
 
 
 def spreadsheet_safe_cell(value: Any) -> Any:
-    """Prefix formula-like text in auditor-facing CSV exports."""
-    if isinstance(value, str) and value.lstrip().startswith(("=", "+", "-", "@")):
-        return "'" + value
+    """Keep untrusted review text visibly textual; source data stays unchanged."""
+    if isinstance(value, str) and value:
+        stripped = value.lstrip()
+        if (value.startswith("text: ") or ord(value[0]) < 32 or ord(value[0]) == 127
+                or value[0] == "\ufeff"
+                or stripped.startswith(("=", "+", "-", "@", "＝", "＋", "－", "＠"))):
+            return "text: " + value
     return value
 
 
+def review_csv_text(frame: pd.DataFrame, *, float_format: str | None = None) -> str:
+    """Write review CSV with LF records and quoted embedded CR/LF unchanged."""
+    exported = frame.map(spreadsheet_safe_cell).rename(columns=spreadsheet_safe_cell)
+    options = {"index": False, "lineterminator": "\r\n", "float_format": float_format}
+    # Using CRLF for each record makes pandas quote both CR and LF inside a
+    # field. Normalize only that record's final terminator, never its cell data.
+    header = exported.iloc[:0].to_csv(**options).removesuffix("\r\n") + "\n"
+    return header + "".join(
+        exported.iloc[index:index + 1].to_csv(header=False, **options).removesuffix("\r\n") + "\n"
+        for index in range(len(exported))
+    )
+
+
 def save_dataframe(df: pd.DataFrame, path: Path) -> Path:
     """Persist a dataframe to parquet (or CSV when the suffix is ``.csv``)."""
     path.parent.mkdir(parents=True, exist_ok=True)

tests/test_reporting.py

@@ -63,10 +63,10 @@ def test_high_risk_extract_escapes_formula_like_text_without_changing_source(
 
     reporting.write_high_risk_extract(vouchers)
     row = pd.read_csv(output, dtype=str, keep_default_na=False).iloc[0]
-    assert row["transaction_id"] == "'" + voucher_id
-    assert row["vendor_name"] == "'  +SUM(1,2)"
-    assert row["description"] == "'-1+2"
-    assert row["risk_reason_text"] == "'@SUM(1,2)"
+    assert row["transaction_id"] == "text: " + voucher_id
+    assert row["vendor_name"] == "text:   +SUM(1,2)"
+    assert row["description"] == "text: -1+2"
+    assert row["risk_reason_text"] == "text: @SUM(1,2)"
     assert row["audit_risk_score"] == "82.0"
     assert vouchers.loc[0, "transaction_id"] == voucher_id
     assert vouchers.loc[0, "vendor_name"] == "  +SUM(1,2)"

tests/test_review_plan.py

@@ -3,6 +3,8 @@
 from __future__ import annotations
 
 import hashlib
+import csv
+import io
 
 import numpy as np
 import pandas as pd
@@ -172,10 +174,10 @@ def test_workpaper_escapes_spreadsheet_formulas_without_changing_selection(popul
     csv_path, json_path = tmp_path / "selection.csv", tmp_path / "selection.json"
     write_review_plan(plan, csv_path, json_path)
     exported = pd.read_csv(csv_path, dtype=str, keep_default_na=False)
-    selected = exported.loc[exported["transaction_id"] == "'" + dangerous_id].iloc[0]
-    assert selected["vendor_id"] == "'  +SUM(1,2)"
-    assert selected["risk_reason_text"] == "'-1+2"
-    assert selected["risk_level"] == "'@SUM(1,2)"
+    selected = exported.loc[exported["transaction_id"] == "text: " + dangerous_id].iloc[0]
+    assert selected["vendor_id"] == "text:   +SUM(1,2)"
+    assert selected["risk_reason_text"] == "text: -1+2"
+    assert selected["risk_level"] == "text: @SUM(1,2)"
     assert selected["selection_reason"].startswith("Covers procedures: ")
     assert plan.queue["transaction_id"].tolist() == original_ids
     assert plan.queue.loc[plan.queue["transaction_id"] == dangerous_id, "vendor_id"].iloc[0] == (
@@ -197,3 +199,20 @@ def test_synthetic_benchmark_is_computed_only_after_the_label_blind_selection(po
     changed["anomaly_label"] = "0"  # parquet also stores labels as strings
     assert build_review_plan(changed, alerts, ReviewPolicy(5, 0.2, 42)).queue.equals(plan.queue)
     assert benchmark_review_plan(plan, changed)["plan_injected_anomalies_found"] == 0
+
+
+def test_multiline_fullwidth_id_survives_workpaper_and_outcome_capture(population, tmp_path) -> None:
+    transactions, alerts = (frame.copy() for frame in population)
+    identifier = "＝1\rsecond\nthird\r\nfourth"
+    transactions.loc[transactions["transaction_id"] == "TX2", "transaction_id"] = identifier
+    alerts.loc[alerts["transaction_id"] == "TX2", "transaction_id"] = identifier
+    plan = build_review_plan(transactions, alerts, ReviewPolicy(5, 0.2, 42))
+    original, record, completed = (tmp_path / name for name in ("original.csv", "record.json", "copy.csv"))
+    write_review_plan(plan, original, record)
+    raw = original.read_bytes()
+    rows = list(csv.DictReader(io.StringIO(raw.decode(), newline="")))
+    assert rows[0]["transaction_id"] == "text: " + identifier
+    assert len(rows) == 5
+    assert plan.queue.loc[0, "transaction_id"] == identifier
+    completed.write_bytes(raw)
+    assert summarize_completed_review(completed, original, record)["selected_count"] == 5

tests/test_utils.py

@@ -9,6 +9,8 @@
 
 from __future__ import annotations
 
+import csv
+import io
 import sys
 from pathlib import Path
 
@@ -20,7 +22,35 @@
 if str(PROJECT_ROOT) not in sys.path:
     sys.path.insert(0, str(PROJECT_ROOT))
 
-from src.utils import REASON_SEPARATOR, as_flag_series, as_list, as_reason_list
+from src.utils import REASON_SEPARATOR, as_flag_series, as_list, as_reason_list, review_csv_text
+
+
+def test_review_csv_marks_formula_variants_and_control_prefixes_as_text() -> None:
+    values = ["=1", "+1", "-1", "@SUM(1)", "＝1", "＋1", "－1", "＠SUM(1)",
+              "  ＝1", "\t=1", "\r=1", "\n=1", "\ufeff=1", "\x00=1", "\x7f=1"]
+    frame = pd.DataFrame({"note": values})
+    rows = list(csv.reader(io.StringIO(review_csv_text(frame), newline="")))
+    assert rows == [["note"], *[["text: " + value] for value in values]]
+    assert frame["note"].tolist() == values
+
+
+def test_review_csv_preserves_multiline_cells_numbers_and_missing_values() -> None:
+    note = 'A\rB\nC\r\nD, "quoted"'
+    frame = pd.DataFrame({"note": [note, "plain"], "amount": [-12.5, np.nan]})
+    rows = list(csv.reader(io.StringIO(review_csv_text(frame, float_format="%.10g"), newline="")))
+    assert rows == [["note", "amount"], [note, "-12.5"], ["plain", ""]]
+
+
+def test_review_csv_prefix_is_collision_safe() -> None:
+    values = ["=1", "text: =1", "text: text: =1"]
+    frame = pd.DataFrame({"note": values})
+    rows = list(csv.reader(io.StringIO(review_csv_text(frame), newline="")))
+    assert rows[1:] == [["text: " + value] for value in values]
+
+
+def test_review_csv_empty_table_still_has_a_safe_header() -> None:
+    text = review_csv_text(pd.DataFrame(columns=["＝field"]))
+    assert list(csv.reader(io.StringIO(text, newline=""))) == [["text: ＝field"]]
 
 
 class TestAsFlagSeries: