Repository navigation
perf: lazy-load the client errors so renderers.configs imports without openai - #199
Merged
Merged
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Approved at Macroscope's review found this PR approvable — The PR narrowly converts two existing client error exports to the package’s established lazy-export mechanism, allowing config-only imports to avoid loading OpenAI while preserving normal access to the errors. It changes only import timing in one module and introduces no schema or production workflow changes. Notes:
You can add or adjust custom eligibility rules. Learn more. |
samsja
reviewed
Oct 22, 2026
| # ``transformers`` dependency is absent. ``__getattr__`` (PEP 562) resolves the | ||
| # names on first attribute access, while ``renderers.base._populate_registry`` | ||
| # handles lazy registration for ``create_renderer``. | ||
| _LAZY_RENDERERS: dict[str, str] = { |
Contributor
Author
There was a problem hiding this comment.
The table now also holds the two renderers.client exceptions, not just renderer classes, so _LAZY_RENDERERS would have been misleading. Happy to rename it back if you prefer.
hallerite
approved these changes
Oct 22, 2026
mikasenghaas
pushed a commit
to PrimeIntellect-ai/verifiers
that referenced
this pull request
Oct 22, 2026
… names lazily (#2938) The config tree is documented to import only types and other configs, but `configs.agent`, `configs.env`, `configs.judge` and `configs.scoring` reached into `clients`, `runtimes` and `interception`, and `verifiers.v1/__init__` imported the whole runtime eagerly. Importing a single config cost ~0.9s and needed openai, anthropic, sympy, numpy, prime-sandboxes and friends installed. The runtime, deployment, interception, tunnel and single-agent env configs now live in `verifiers.v1.configs` (old import paths keep re-exporting them), `types.py` imports numpy only where it builds arrays, and `verifiers.v1` resolves its names on first access through one name to module table, with the same imports under `TYPE_CHECKING` so type checkers still see real types. The whole config tree now imports with only pydantic, in ~140ms. No public API change: every `vf.*` name and existing import path still resolves. `tests/unit/test_api.py` keeps the lazy table, the `TYPE_CHECKING` imports and `__all__` in sync and asserts the config tree imports without the runtime; that test needs PrimeIntellect-ai/renderers#199 released and the renderers pin bumped, which I'll add here once it's published. Unit tests and `ty check` pass. prime-rl's 29 example RL/SFT configs validate in an 11-package env with output identical to a full install. <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Keep `verifiers.v1` config tree free of runtime imports and resolve names lazily > - Moves all config models (`RuntimeConfig`, `InterceptionConfig`, `DeploymentConfig`, `SingleAgentEnvConfig`) from runtime/interception/env implementation modules into a dedicated `verifiers.v1.configs` package, so importing the config tree no longer pulls in runtime dependencies > - Makes [__init__.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2938/files#diff-731a849803c53d001c41b5e0fb1bfcb392eaae7eb4fa4b3ace382768dc40066a) import public symbols only under `TYPE_CHECKING` and resolves them at attribute access via a new `__getattr__` registry that caches resolved values; `__dir__` lists unresolved lazy names > - Moves the NumPy import in [types.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2938/files#diff-88a4e4ee0c58ac5dc7344161a011374082a3767eda2ad32af97c942dca982112) to call time inside array-building helpers, so `verifiers.v1.types` imports without NumPy > - Adds tests in [test_api.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2938/files#diff-3cabff7688a6836f0be39ae71c04a68b09858683669c2bf6d24281910999cb74) that the lazy mapping matches the type-checking imports, `__all__` stays in sync, and every config module imports without runtime packages > - Behavioral Change: `dir(verifiers.v1)` and import-time side effects differ — the config models now live only under `verifiers.v1.configs` (e.g. `DeploymentConfig`, `PrimeConfig`); old import paths like `verifiers.v1.runtimes.PrimeConfig` are removed, and in-tree importers were updated to the new modules > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized c289d0c.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here -->
mikasenghaas
added a commit
to PrimeIntellect-ai/verifiers
that referenced
this pull request
Oct 22, 2026
## Summary - Bump `renderers[multimodal]` from `>=0.1.12` to `>=0.1.13.dev16`, and lock 0.1.13.dev16. - dev16 is the build of PrimeIntellect-ai/renderers#199, which lazy-loads the client errors. With 0.1.12, `from renderers import RendererConfig` in `configs/client.py` loads `openai`, `httpx` and `rich`. - I locked with uv 0.12.6 to keep the lockfile format unchanged, so the lock diff only touches renderers. ## Verification - Before: `test_config_tree_imports_without_the_runtime` fails on `main` (`7f3290501`) in CI and locally. The extra packages are `openai`, `httpx`, `rich`, `anyio`, `pygments` and others. - After: the same test passes, and the full unit suite passes locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Dependency-only bump with no application code changes; risk is limited to behavior differences in the pinned pre-release renderers build. > > **Overview** > Bumps **`renderers[multimodal]`** from `>=0.1.12` to **`>=0.1.13.dev16`** and refreshes **`uv.lock`** to pin that pre-release (renderers#199). > > The newer build **lazy-loads client/OpenAI-related code**, so importing the config tree (e.g. `RendererConfig` via `verifiers.v1.configs`) no longer drags in **`openai`**, **`httpx`**, **`rich`**, and their transitive deps. That restores **`test_config_tree_imports_without_the_runtime`**, which expects only a small allowlist of top-level packages when walking `verifiers.v1.configs`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit aeee131. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Bump `renderers[multimodal]` to `0.1.13.dev16` so config tree imports without openai > Raises the minimum `renderers[multimodal]` dependency from `0.1.12` to `0.1.13.dev16` in [pyproject.toml](https://github.com/PrimeIntellect-ai/verifiers/pull/2950/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711). Updates [uv.lock](https://github.com/PrimeIntellect-ai/verifiers/pull/2950/files#diff-84321598744d84dbee2318e634c74c9aae39a1c253f1c4bd17ebf9ef2f807b11) to match. Risk: installs now require a dev prerelease (`0.1.13.dev16`) rather than a stable `renderers` release. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized aeee131.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
mikasenghaas
added a commit
to PrimeIntellect-ai/prime-rl
that referenced
this pull request
Oct 22, 2026
…t runs (#4206) ## Summary - Bump `deps/verifiers` to verifiers `main` (`4c2738dc6`), which includes PrimeIntellect-ai/verifiers#2949. Its new agent config fields reach prime-rl through each source's `env` block: - `env.agent.scoring.skip_on_truncated`: skip task scoring for a truncated trace. It keeps no rewards, so its reward is 0. - `env.agent.scoring.skip_on_timeout`: the same for a trace whose stage deadline expired before scoring. - `env.agent.max_turn_tokens`: the framework per-turn limit. It lowers `sampling.max_tokens` on every request. Like the other `max_*` caps, it stops the run once a response hits it, with the `max_turn_tokens` stop condition. This also stops harnesses that continue past a capped turn, such as RLM. - Count `max_turn_tokens` as truncation in both copies of `TRUNCATING_STOPS`, which mirror `Trace.is_truncated`: `monitors/file/traces/index.py` and the dashboard's `static/app.js`. - Bump `deps/renderers` to renderers `main` (PrimeIntellect-ai/renderers#199, 0.1.13.dev16), because verifiers `main` now requires `renderers>=0.1.13.dev16` (PrimeIntellect-ai/verifiers#2950). - The verifiers bump also moves `prime-sandboxes` 0.4.7 → 0.5.0 in `uv.lock`, because verifiers `main` requires it. - Mention the flags in the eval skill. ```toml [[source]] env.taskset.id = "gsm8k" env.agent.max_turn_tokens = 4096 env.agent.scoring.skip_on_truncated = true env.agent.scoring.skip_on_timeout = true ``` ## Verification I ran `uv run eval` on Prime Inference (default model), null harness, subprocess runtime. - **Turn cap**, alphabet-sort, `max_turns = 1`, 4 tasks × 2: - Skip off: the two `max_turns`-stopped traces score `alphabet_sort: 0.5`. - Skip on: the same traces get `rewards={}` (reward 0) and `info.scoring_skipped="truncated"`. Traces that end normally still score. - **Token cap**, gsm8k, `max_completion_tokens = 24`, 8 tasks × 2: - Skip off: all 16 traces finish with `length` and score `correct: 0.0`. - Skip on: all 16 get `rewards={}` and `scoring_skipped="truncated"`. - **Per-turn limit**, alphabet-sort, skip on: - `max_turn_tokens = 8`: every request carries `max_tokens=8`, and the two-turn tasks end after 1 call with `stop=max_turn_tokens`. - `sampling.max_tokens = 8` instead: the two-turn tasks make 2 capped calls. - Every trace is truncated and keeps no rewards. - Unit tests in `tests/unit/{eval,orchestrator}` and the dashboard and config tests: 409 pass. One test needs `vllm` and one needs `prime_kernels`. Neither is installed on macOS. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Low risk: truncation classification parity in monitors/UI, eval docs, and dependency lock updates; scoring behavior changes come from the verifiers bump rather than new prime-rl logic in this diff. > > **Overview** > Aligns prime-rl with verifiers’ **per-turn token cap** and **optional zero-reward scoring** when runs truncate or time out before scoring. > > **Truncation handling:** `max_turn_tokens` is added to `TRUNCATING_STOPS` in the dashboard (`app.js`) and file trace monitor (`traces/index.py`) so traces stopped by the per-turn limit are treated like other framework caps when mirroring `Trace.is_truncated` (e.g. for `is_truncated/mean` and UI). > > **Docs:** The eval skill now explains `env.agent.max_turn_tokens` and `env.agent.scoring.skip_on_truncated` / `skip_on_timeout` as alternatives to raising `sampling.max_completion_tokens` when truncation metrics are high. > > **Dependencies:** `uv.lock` bumps `prime-sandboxes` to 0.5.0 and `renderers` to `>=0.1.13.dev16` (with verifiers), plus lock revision. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit d06f443. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
renderers/__init__eagerly importedrenderers.clientfor its two exceptions, so importing only the config layer (renderers.configs, which verifiers and prime-rl's config schema use) also loaded httpx and openai. The two exceptions now resolve through the existing PEP 562 table alongside the lazy renderer classes, with aTYPE_CHECKINGimport so their types stay visible.import renderers.configsnow loads only pydantic.No API change:
from renderers import OverlongPromptErrorstill works. Tested withtests/test_client.py, ruff and ty (same diagnostics as main), and an import check thatrenderers.clientandopenaistay out ofsys.modules. This unblocks a pydantic-only environment for hosted training's config validation (paired verifiers and prime-rl PRs).Note
Lazy-load client error classes in
rendererssorenderers.configsimports without openaiRemoves the eager import of HTTP client error classes from renderers/init.py. They are instead added to the lazy export registry and resolved on demand via
renderers.__getattr__, which importsrenderers.clientand caches the result in package globals.dir(renderers)now lists the lazily exported client error class names (MalformedGenerateResponseError,OverlongPromptError) alongside renderer classes.Macroscope summarized 1a3b5d2.