Skip to content

astrta medium: Separate validation progress from solver performance benchmarks - #6

Merged
TiloRC merged 4 commits into
TiloRC:mainfrom
tilorc-bot:fix/validation-performance
Sep 22, 2026
Merged

TiloRC merged 4 commits into
TiloRC:mainfrom
tilorc-bot:fix/validation-performance

Conversation

@tilorc-bot

Copy link
Copy Markdown
Contributor

astrta medium:

Validation wall time included an expensive SymPy generated-facts check that never entered either substituted backend. It also measured a changing workload: fixing an early assertion lets a test execute more queries. This change makes correctness progress and fixed-workload performance independently trackable, so optimization can proceed alongside support for more cases.

  • Exclude only test_query::test_known_facts_consistent from routine backend comparisons and ASV progress runs, using normalized test identities. --include-controls and plain pytest validation retain the full upstream suite; other slow tests remain selected.
  • Preserve per-test outcomes and failure details with --output, uploaded by CI. Version the pass-count series for the 104-to-103 population change and rename the wall-time series to validation execution cost.
  • Add checked ASV timings for the full pipeline, discovery/encoding, engine construction, first-query solving, and sum scaling. Workloads combine existing examples with individual pinned-upstream assertions. Actual-answer tracking remains available when a timing's answer contract fails. Warm-cache policy and fresh solver state are explicit.
  • Configure the hash seed using ASV's supported environment matrix, and add a tested container workflow for CPU-pinned measurements and saved results tied to the tested commit.

Validation:

  • Unit and harness tests: 157 passed, 1 xfailed; strict mypy and vendored-facts regeneration check passed.
  • ASV 0.6.6 discovery/check and all 14 benchmark definitions (70 parameter combinations) passed a container smoke run. Repeated solve and scaling measurements were also successfully saved against the tested commit.
  • Full versus filtered validation ran under one CPU-10/domain lease with hash seed 0: 104 → 103 tests per backend, every other outcome identical. Observed validation execution times were reasoning 50.82 → 4.94 s and SymPy 51.85 → 5.13 s. These are single-run harness-cost observations, not solver speedup claims.
  • Regression tests cover targeted exclusion for both wrapper paths, retention of unrelated slow tests, collection failures, equal pass totals hiding different outcomes, answer loss, and fresh engine state.

No solver implementation or upstream test bodies change. Optional backend-only timing instrumentation is unnecessary for the fixed-workload performance track and is not added.

Implements the assessment and requested changes.

Closes tilorc-bot#1

@TiloRC
TiloRC merged commit 95f1ae8 into TiloRC:main Sep 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark hygiene: validation timing metrics are dominated by a backend-independent SymPy test

2 participants