Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,25 @@ Unknown, duplicate or malformed view parameters show a notice and the latest
configured panel. With JavaScript disabled, all six demo panels remain readable.
The report links to its method, GitHub project and author for further review.

## Download one view for review

Each panel has **Download these metrics · CSV**. It saves only that cutoff/year's
nine metric rows, including the formula, unit, status, exact fraction, rounded
display and missing-input names. Switching views or reopening a shared link
selects the matching panel's download. **All metrics CSV** still contains all
54 demo rows. Every panel's download also works with JavaScript disabled.

For Excel, use **Data → From Text/CSV**, select UTF-8, and keep the `as_of`,
`period` and `exact` columns as text to preserve dates, identities and fractions.
The `display` column is rounded; `exact` retains the saved rational value.
The CSV uses the same text markers for formula-shaped labels as the full export.

Try the before/after-filing links above, downloading each view. FY2024 before
filing has nine `missing_input` rows with empty values. After filing,
`free_cash_flow` retains `108807000000/1`, unit `USD`, status `ok` and the
`operating_cash_flow - capex` formula. This is a filtered review export;
full source/lineage review and semantic verification use the complete report.

## Contracts

| Concern | Behavior |
Expand Down
6 changes: 6 additions & 0 deletions docs/INTERVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@
先指出前一日缺失不等于零,再展开后一日的自由现金流 **108,807 百万美元**及其输入与 SEC 来源。
这里使用手工整理的公开年报摘录与 SEC 申报日期,不是原样 API 下载或盘中可得时间证明。

再用两分钟练一次实际交付:在两种截止日下分别点 **Download these metrics · CSV**,
核对每份只有九行,且 `as_of`、`period` 与页面一致。申报前的 `exact`、`display` 留空,
状态是 `missing_input`;申报后 FCF 为 `108807000000/1` USD,仍保留公式。
Excel 用“数据 → 从文本/CSV”按 UTF-8 导入,将 `as_of`、`period`、`exact` 设为文本,
再解释精确值、显示值与缺失值的区别。指标摘录须连同完整报告的事实和血缘继续审阅。

**验证**:独立 SQL 查询逐条对照。修改指标 CSV 并更新哈希,语义复算仍拒绝。测试覆盖未来信息、季度混入、单位不匹配、同日冲突、缺失值与零分母。

追问要点:`fy` 不能代替事实本身的期间;缺失不等于零;增长没有按周数归一化;哈希验证一致性而不证明来源真实性;扩展到其他公司需要补充概念、单位与会计政策映射。
Expand Down
25 changes: 25 additions & 0 deletions docs/METHODOLOGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,28 @@ reports should be regenerated into a new directory from their saved facts and
request before verification with this renderer.
Pages deployment also waits for this browser job, alongside the Linux/Windows
test and installed-wheel checks.

## Current-panel metric downloads

Each saved panel embeds a UTF-8 CSV data URL with exactly its metric rows and
the same nine columns as the full `metrics.csv`. Both use the same CSV serializer
and text-label contract. Full fractions, negative numbers, formulas, statuses
and empty missing values remain unchanged; the browser does not scrape displayed
numbers or recalculate financial values. Filenames contain the validated cutoff
date; original period identities stay in the rows rather than the filename.

The native download anchor belongs to its panel, so link/history restoration
and invalid-link fallback expose the corresponding download. Without JavaScript,
all panels retain their own links. CSV importers may infer cell types; select
UTF-8 and import `as_of`, `period` and `exact` as text when using Excel.
This excerpt alone is not a complete replay bundle: source facts, lineage and
verification remain available from the report's full nine-file inventory.

Unit checks compare every panel's decoded download with its full-export slice,
including formula-shaped labels, controls/Unicode, quoted newlines, negative
values and hand-checked before/after-filing results. Real Chrome saves actual
files, parses their rows independently and compares every column with the full
export, including after view changes/fallback and without JavaScript. Browser
checks assert the actual `innerWidth` at desktop/narrow sizes. Only committed
HTML and its manifest hash are refreshed; seven financial/input files retain
their original bytes.
2 changes: 1 addition & 1 deletion examples/demo/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion examples/demo/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"analysis.json": "ae9fe1aec6d4b683ca15ec9007ea14eca32d63a4c15f26e88cac5308b43aaac5",
"decisions.csv": "cebbcc9cc7a7bd25735cedb88fad80151b76f57fffb1bcd687666a75d796bdf3",
"facts.json": "93d636674792ae8f5cffdbae4c7cf6e27ffa4ac003991fb0976d3e0225c9c8f5",
"index.html": "e40a8b859316a9ee4d17744ce15821ff4578fea11051422324a0b22b0bc60701",
"index.html": "ae1f9ec959ae1d4f339589d4024e70adcd4fa2646a0ba22feb26dbaa2797642b",
"lineage.csv": "bba7e396cb6159d1cc786d4c77e288a33e5f62d2b32bedf2c5dca203f852e74f",
"metrics.csv": "a0d996134f4e68bade395f7cde6bd3e51a53b6bd8723b3bb67e9027413fc08a7",
"request.json": "042942ccc50dbf51be6f593213f8b9f47276bcd36026a62735946a3f5d1fe3ab",
Expand Down
80 changes: 74 additions & 6 deletions scripts/check_browser.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
CHROMEWEBDRIVER directory supplied by GitHub's Ubuntu runner image.
"""

import csv
import io
import json
import os
import shutil
Expand All @@ -26,6 +28,10 @@
if (JSON.stringify(values) !== JSON.stringify(expected.metrics)) throw new Error('Financial metrics differ');
const sources = Array.from(selected[0].querySelectorAll('table')[1].querySelectorAll('a'), a => a.href);
if (JSON.stringify(sources) !== JSON.stringify(expected.sources)) throw new Error('Source links differ');
const download = selected[0].querySelector('.view-csv');
if (!download || download.download !== 'statementtrace-metrics-' + expected.as_of + '.csv' ||
!download.href.startsWith('data:text/csv;charset=utf-8;base64,'))
throw new Error('Missing current-view CSV download');
const link = document.getElementById('view-link');
const url = new URL(link.href);
const params = new URLSearchParams(url.hash.slice(1));
Expand Down Expand Up @@ -94,6 +100,8 @@ def call(method, endpoint, payload=None):
except urllib.error.HTTPError as error:
raise RuntimeError(error.read().decode()) from error

downloads = root / "downloads"
downloads.mkdir()
session = None
process = subprocess.Popen([driver, f"--port={port}"], stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
try:
Expand All @@ -114,7 +122,13 @@ def call(method, endpoint, payload=None):
"capabilities": {
"alwaysMatch": {
"browserName": "chrome",
"goog:chromeOptions": {"args": ["--headless=new", "--no-sandbox", "--disable-dev-shm-usage"]},
"goog:chromeOptions": {
"args": ["--headless=new", "--no-sandbox", "--disable-dev-shm-usage"],
"prefs": {
"download.default_directory": str(downloads),
"download.prompt_for_download": False,
},
},
}
}
},
Expand All @@ -133,19 +147,60 @@ def wait_view(view, *, exact=True, unavailable=False):
time.sleep(0.05)
return execute(CHECK_VIEW, view, exact, unavailable) # Preserve the actual assertion diagnostic.

counts = dict(visits=0, selections=0, metric_rows=0, round_trips=0, history_checks=0, invalid_links=0)
def download_view(view, selector="section:not([hidden]) .view-csv"):
path = downloads / f"statementtrace-metrics-{view['as_of']}.csv"
path.unlink(missing_ok=True)
element = call("POST", f"/session/{session}/element", {"using": "css selector", "value": selector})[
"element-6066-11e4-a52e-4f735466cecf"
]
call("POST", f"/session/{session}/element/{element}/click", {})
for _ in range(100):
if path.is_file():
break
time.sleep(0.05)
else:
raise AssertionError("Chrome did not save the view CSV")
raw = path.read_bytes()
rows = list(csv.DictReader(io.StringIO(raw.decode("utf-8"), newline=""), strict=True))
if rows != view["csv_rows"]:
raise AssertionError("Downloaded CSV differs from the selected full-export slice")
counts["downloads"] += 1
counts["downloaded_metric_rows"] += len(rows)

counts = dict(
visits=0,
selections=0,
metric_rows=0,
round_trips=0,
history_checks=0,
invalid_links=0,
downloads=0,
downloaded_metric_rows=0,
)
viewport_widths = []
for width in (1280, 390):
call("POST", f"/session/{session}/window/rect", {"width": width, "height": 1000})
for case in cases:
for case_index, case in enumerate(cases):
base = (root / case["file"]).as_uri() + "?review=1"
navigate(base)
actual_width = execute("return window.innerWidth;")
if actual_width != width:
raise AssertionError(f"Requested width {width}, observed {actual_width}")
if case_index == 0:
viewport_widths.append(actual_width)
counts["visits"] += 1
latest = case["views"][-1]
wait_view(latest, exact=False)
for index, view in enumerate(case["views"]):
execute(SELECT_VIEW, view, "cutoff" if index % 2 else "period")
counts["selections"] += 1
counts["metric_rows"] += len(view["metrics"])
download_view(latest)
before = next(
v for v in case["views"] if v["as_of"] == "2024-10-31" and v["period"] == latest["period"]
)
execute(SELECT_VIEW, before, "cutoff")
download_view(before)
first = case["views"][0]
shared = execute(SELECT_VIEW, first, "period")
navigate(base)
Expand Down Expand Up @@ -177,6 +232,8 @@ def wait_view(view, *, exact=True, unavailable=False):
]
call("POST", f"/session/{session}/element/{element}/click", {})
wait_view(latest)
if case_index == 0:
download_view(latest)

call(
"POST",
Expand All @@ -188,6 +245,7 @@ def wait_view(view, *, exact=True, unavailable=False):
visible = execute("return document.querySelectorAll('section:not([hidden])').length;")
if visible != len(cases[0]["views"]) or not execute("return document.getElementById('share-view').hidden;"):
raise AssertionError("No-JavaScript view must retain every saved panel")
download_view(cases[0]["views"][0], "section .view-csv")
finally:
call(
"POST",
Expand All @@ -200,7 +258,7 @@ def wait_view(view, *, exact=True, unavailable=False):
"status": "passed",
"browser": "Chrome",
"reports": len(cases),
"viewports": [1280, 390],
"viewports": viewport_widths,
"no_js_panels": visible,
**counts,
}
Expand All @@ -220,13 +278,14 @@ def wait_view(view, *, exact=True, unavailable=False):
def main():
from statementtrace.cli import demo_inputs
from statementtrace.engine import analyze
from statementtrace.report import build
from statementtrace.report import build, csv_label

pairs = [
["2023", "2024"],
["sample\rone", "sample\none"],
["sample\x00one", "sample�one"],
[' 年报 +&%#?/ " 🧾 ', "\tAnnual <2024>\n"],
["=1+1", "text: =1+1"],
]
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary).resolve()
Expand All @@ -236,13 +295,22 @@ def main():
for row, label in zip(request["periods"], labels, strict=True):
row["label"] = label
filename = f"report-{index}.html"
(root / filename).write_bytes(build(data, request)["index.html"])
outputs = build(data, request)
(root / filename).write_bytes(outputs["index.html"])
all_metrics = list(
csv.DictReader(io.StringIO(outputs["metrics.csv"].decode("utf-8"), newline=""), strict=True)
)
views = [
{
"as_of": panel["as_of"],
"period": panel["period"]["label"],
"metrics": [[m[key] for key in ("metric", "display", "unit", "status")] for m in panel["metrics"]],
"sources": [fact["source_url"] for fact in panel["facts"].values() if fact is not None],
"csv_rows": [
row
for row in all_metrics
if row["as_of"] == panel["as_of"] and row["period"] == csv_label(panel["period"]["label"])
],
}
for panel in analyze(data, request)["panels"]
]
Expand Down
32 changes: 23 additions & 9 deletions src/statementtrace/report.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Self-contained HTML, CSV, and semantic replay of a deterministic report."""

import base64
import csv
import html
import io
Expand All @@ -20,6 +21,8 @@
"index.html",
}

METRIC_FIELDS = ("as_of", "period", "metric", "formula", "unit", "status", "exact", "display", "missing")

VIEW_SCRIPT = r"""
const cutoff = document.querySelector('#cutoff');
const period = document.querySelector('#period');
Expand Down Expand Up @@ -108,6 +111,18 @@ def csv_label(value):
return value


def metric_rows(panel):
return [
{
"as_of": panel["as_of"],
"period": panel["period"]["label"],
**{key: metric[key] for key in ("metric", "formula", "unit", "status", "exact", "display")},
"missing": ";".join(metric["missing"]),
}
for metric in panel["metrics"]
]


def render(analysis):
esc = html.escape
panels = []
Expand Down Expand Up @@ -135,12 +150,17 @@ def render(analysis):
)
label, cutoff = esc(panel["period"]["label"]), esc(panel["as_of"])
period_key = esc(json.dumps(panel["period"]["label"], ensure_ascii=True), quote=True)
metric_csv = base64.b64encode(csv_bytes(metric_rows(panel), METRIC_FIELDS)).decode("ascii")
panels.append(
f"<section data-period='{period_key}' data-cutoff='{cutoff}'>"
f"<h2>FY {label} · as of {cutoff}</h2><p>"
f"{esc(panel['period']['start'])} → {esc(panel['period']['end'])} · "
f"{panel['duration_days']} inclusive days · amounts in USD</p>"
"<h3>Financial metrics / 财务指标</h3><div class='scroll'><table>"
"<h3>Financial metrics / 财务指标</h3>"
f"<p><a class='view-csv' href='data:text/csv;charset=utf-8;base64,{metric_csv}' "
f"download='statementtrace-metrics-{cutoff}.csv'>Download these metrics · CSV</a>"
" · Includes this cutoff/year, formulas, exact values and missing-input statuses.</p>"
"<div class='scroll'><table>"
"<thead><tr><th>Metric</th><th>Value</th><th>Unit</th><th>Status</th>"
"<th>Formula and lineage</th></tr></thead><tbody>" + "".join(metrics) + "</tbody></table></div>"
"<h3>Selected facts / 原始数据</h3><div class='scroll'><table>"
Expand Down Expand Up @@ -191,7 +211,7 @@ def options(values, *, identities=False):
"<p id='view-status' role='status'></p>"
"<noscript><p>JavaScript is off; all saved cutoff/year panels are shown below.</p></noscript>"
"<p><a href='selected_facts.csv'>Facts CSV</a> · "
"<a href='metrics.csv'>Metrics CSV</a> · <a href='lineage.csv'>Full lineage</a> · "
"<a href='metrics.csv'>All metrics CSV</a> · <a href='lineage.csv'>Full lineage</a> · "
"<a href='decisions.csv'>Selection decisions</a> · "
"<a href='manifest.json'>SHA-256 manifest</a></p></div>"
+ "".join(panels)
Expand Down Expand Up @@ -225,14 +245,8 @@ def build(data, request):
"source_url": fact["source_url"] if fact else "",
}
)
metrics.extend(metric_rows(panel))
for metric in panel["metrics"]:
metrics.append(
{
**base,
**{key: metric[key] for key in ("metric", "formula", "unit", "status", "exact", "display")},
"missing": ";".join(metric["missing"]),
}
)
lineage.extend({**base, "metric": metric["metric"], **item} for item in metric["inputs"])
files = {
"facts.json": canonical(data),
Expand Down
Loading
Loading