Skip to content

[feat] Configure the context window for custom OpenAI-compatible models #150

Description

@cenab

[feat] Configure the context window for custom OpenAI-compatible models

What problem are you trying to solve?

At 4585c8e455ac5bdeb76b618343ce748ef08134f4, a custom OpenAI-compatible
endpoint can be selected with api_provider, base_url, and api_key_helper,
but crates/cli/src/agent.rs constructs every SessionConfig with
context_window: 200_000. A custom endpoint with a 32,768-token window therefore
cannot declare its actual context budget through the same settings.

The existing provider works: an external Rust test runner used
crab_api::create_backend with the ordinary configuration and streamed requests
through the native OpenAI serializer and SSE parser for tsubasa-fast and
tsubasa-pro. Four controlled requests covered text and HTTP 401, with a
4,096-token output request. Each outgoing body passed the real Tsubasa strict
schema and input/output budget check before a fixture response. The existing
38 OpenAI tests also pass. These checks do not establish a full CLI session or
live model qualification; the fixed session window is a separate source finding.

Proposed solution

Add an optional context-window setting for a selected custom model and pass it
to SessionConfig. Preserve the current default when no override is present.
The same value should govern compaction/admission, including output headroom.
Tests could load the override through normal settings and verify that session
construction uses 32,768, while an unconfigured model preserves existing behavior.

Alternatives considered

Reducing --max-tokens bounds output but does not correct a 200,000-token
session context assumption. A new provider or custom transport would duplicate
the already-working OpenAI path.

Prior art

The existing custom-provider settings establish the endpoint-selection route.
Related settings issue #6 and provider proposal #128 were reviewed; this request
specifically concerns the session context limit.

Scope

LLM provider, Settings, Session, CLI.

Are you willing to contribute?

  • I'm willing to submit a PR.
  • I can help test.

Prepared with AI assistance on macOS arm64 with Rust 1.98.1. No credentials or
private prompts are included.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions