Skip to content

Add interactive model experiment suites and prepare 0.1.1 - #2

Merged
jpn-- merged 4 commits into
mainfrom
feature/data-assets
Sep 19, 2026
Merged

jpn-- merged 4 commits into
mainfrom
feature/data-assets

Conversation

@jpn--

@jpn-- jpn-- commented Sep 19, 2026

Copy link
Copy Markdown
Member

Changes

Model repositories can now provide self-contained benchmark suites that download checksum-verified data through ActivitySim's shared cache, resolve a current GitHub branch or PR head to an exact source commit, and prompt for typed experiment inputs.

abench /path/to/model discovers YAML suites in .abench, asks which experiment to run, then collects its inputs. Defaults can be accepted with Enter; required or invalid values prompt again. Explicit --set values and --non-interactive support automation. Suite metadata records the original instructions, effective inputs, and resolved commits.

Long builds and model phases now print periodic elapsed-time updates, with current and peak cgroup memory when available. Full logs remain on disk.

Prepares release 0.1.1 and updates the documentation and SANDAG example.

Validation

  • 146 unit tests passed locally.
  • Pre-commit lint and formatting checks passed.
  • Four Docker integration tests passed during implementation, covering serial/MP runs, cache retries, shared data preparation, and typed overrides.
  • Interactive input and directory selection verified in a real terminal.
  • GitHub CI checks Python 3.10–3.12, built wheels, and Docker integration before release.

Support --set overrides with validation and file-specific help. Resolve branch and PR sources to exact commits, prepare checksum-verified model assets in shared caches, and record suite provenance.
@jpn--
jpn-- merged commit d09b0e7 into main Sep 19, 2026
8 checks passed
@jpn--
jpn-- deleted the feature/data-assets branch September 19, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant