Skip to content

Serve hgvs UTA lookups from a recorded cache in tests - #667

Open
theferrit32 wants to merge 10 commits into
mainfrom
kf/659-hgvs-uta-cache
Open

theferrit32 wants to merge 10 commits into
mainfrom
kf/659-hgvs-uta-cache

Conversation

@theferrit32

@theferrit32 theferrit32 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Closes #659.

Tests no longer connect to the public UTA database or fetch sequences from NCBI. The hgvs library's data provider lookups (UTA queries and sequence fetches) are replayed from a recorded cache, tests/data/hgvs_cache.json, using hgvs's built-in cache modes. Library code is unchanged; the cache is configured only in the test suite.

  • A session fixture in tests/conftest.py gives every HgvsTools one shared, cache-backed hgvs data provider. VRS_HGVS_CACHE_MODE selects run (default, never opens a UTA connection), learn, or verify, or empty to use UTA directly. These variables are read only by the test suite.
  • hgvs stores its cache as a pickle, which can run code when loaded and can't be reviewed in a diff. The cache is committed as JSON instead, and tests/hgvs_cache_json.py converts it to a temporary pickle for hgvs when tests start.
  • make record-hgvs-cache re-records the cache against UTA and a seqrepo-rest-service: hgvs records into the pickle named by VRS_HGVS_CACHE_FILE (learn mode only), and the JSON is written only if the test run passes. VCR cassettes are not touched.
  • Re-recorded the hgvs-related cassettes, dropping NCBI requests that are now served from the cache.
  • A test checks that a full hgvs translation in run mode makes no UTA queries or sequence fetches.
  • The README now describes recording both VCR cassettes and the hgvs cache.

In CI the pytest step drops from about 60 s to about 10 s per Python version, and tests no longer hang or fail when the public UTA server is unavailable.

Add VRS_HGVS_CACHE_MODE/VRS_HGVS_CACHE_FILE to enable the hgvs data provider cache,
default tests to run mode with tests/data/hgvs_cache.pkl, and add
`make record-hgvs-cache` to re-record it against UTA and a seqrepo-rest-service.
hgvs sequence fetches are now served from tests/data/hgvs_cache.pkl, so the
NCBI E-utilities interactions recorded in these cassettes are no longer used.
@theferrit32
theferrit32 marked this pull request as ready for review October 6, 2026 04:37
@theferrit32
theferrit32 requested review from a team as code owners October 6, 2026 04:37
Comment thread tests/conftest.py Outdated
@theferrit32
theferrit32 marked this pull request as draft October 6, 2026 20:04

@theferrit32 theferrit32 Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously I was just checking in the pickle file that biocommons/hgvs creates. But that has some security concerns because it's binary serialized python that cannot be easily inspected in a pr/commit diff and can execute arbitrary code when deserialized. So now this declarative json form gets written instead, and conftest.py converts it into the pickle form that biocommons/hgvs expects, on the fly.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The documented HgvsTools environment-variable behavior is implemented only by the pytest fixture.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
What changed in this PR

Adds a recorded HGVS data-provider cache so tests can run without live UTA or NCBI access.

Changes:

  • Adds JSON-to-pickle HGVS cache conversion and a shared cached provider fixture.
  • Adds offline-cache coverage and removes superseded NCBI cassette interactions.
  • Documents and automates cache re-recording.
File Description
.gitignore Ignores interrupted cache recordings.
Makefile Adds the cache recording target.
README.md Documents recorded test data workflows.
tests/​conftest.py Configures the shared HGVS cache provider.
tests/​hgvs_cache_json.py Converts HGVS caches between JSON and pickle.
tests/​data/​hgvs_cache.json Stores recorded UTA and sequence results.
tests/​extras/​test_hgvs_tools.py Verifies provider sharing.
tests/​extras/​test_allele_translator.py Tests offline HGVS translation.
tests/​extras/​cassettes/​test_hgvs_cache_run_mode_is_offline.yaml Records local SeqRepo interactions.
tests/​extras/​cassettes/​test_rle_seq_limit.yaml Removes cached NCBI requests.
tests/​extras/​cassettes/​test_hgvs[NM_181798.1:c.1007G>T-expected7].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NM_001331029.1:c.722A>G-expected6].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000019.10:g.289485_289500del-expected9].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000019.10:g.289464_289465insCACA-expected8].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000013.11:g.32936732=-expected0].yaml Removes an NCBI sequence fetch.
tests/​extras/​cassettes/​test_hgvs[NC_000013.11:g.32331093_32331094dup-expected4].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000013.11:g.32316467dup-expected5].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000007.14:g.55181320A>T-expected1].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000007.14:g.55181230_55181231insGGCT-expected3].yaml Removes NCBI sequence fetches.
tests/​extras/​cassettes/​test_hgvs[NC_000007.14:g.55181220del-expected2].yaml Removes NCBI sequence fetches.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/conftest.py
@theferrit32
theferrit32 marked this pull request as ready for review October 7, 2026 16:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(ci): dont make live network requests

2 participants