Skip to content

feat(web): configure generation recovery when deploying models - #78

Merged
imneov merged 2 commits into
modelsphere:mainfrom
imneov:feat/63-generation-recovery
Nov 6, 2026
Merged

imneov merged 2 commits into
modelsphere:mainfrom
imneov:feat/63-generation-recovery

Conversation

@imneov

@imneov imneov commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • feature
  • documentation

What this PR does and why

Model Serving has no form controls for continuation_gateway. Add generation recovery to deploy and upgrade with actual model identity, stream idle timeout, continuation execution, and tool-call buffering, using the existing chart values and plan API.

Rebased on current main while preserving adaptive-concurrency cleanup, cache settings, apply flags, and the upgrade form's stable initial snapshot. Validate the effective plan across its configuration layers and report higher-priority YAML that overrides the recovery controls. Clearing an upgrade's timeout removes the stored form override through a fresh composition pinned to the validated source and chart; unrelated changes stop review. Existing detection-only mode and unrelated gateway settings remain supported.

Which issue(s) this PR fixes

Fixes #63

How it was tested

  • pnpm run typecheck: Console and standalone Swiss types pass.
  • pnpm exec vitest run: 58 files, 520 tests pass, including 31 recovery-related tests.
  • Regression tests reproduced three validation failures before their fixes. Timeout-reset tests also failed against the previous pass-through composition, then passed after implementing removal.
  • Tests cover fromRelease retaining a cleared timeout, source/chart/unrelated-value drift, higher-priority YAML, later plan layers, detection-only mode, and simultaneous adaptive-concurrency and timeout cleanup.
  • pnpm run build and pnpm run build:swiss pass.
  • go vet ./..., go test ./..., Go formatting, and git diff --check pass.
  • Agent-browser checked the real shared form components with sample settings. Preview screenshots are kept locally, outside the repository. No live model deployment or crash recovery was performed.

Checks

  • go test ./... and gofmt -l . is empty
  • Frontend typecheck, tests, and production builds
  • helm lint --strict helm/console (no chart changes)
  • English/Chinese guides and design documentation updated
  • No internal hostnames, IPs, registries or credentials

Notes for reviewers

Start with recovery-compose.ts, recovery.ts, and their tests. Recovery and adaptive-concurrency cleanup share the source-pinning and baseline comparison in plan-composition.ts. The copied Swiss module is unchanged. Chart compatibility remains sglang 0.8.11+ or vllm 0.8.9+; operators must verify their engine/model combination with the gateway documentation.

Release note

Model Serving deploy and upgrade forms can configure generation recovery, including actual model identity, idle timeout, detection-only mode and tool-call buffering.

@imneov
imneov force-pushed the feat/63-generation-recovery branch 2 times, most recently from 52e576f to 9e57a98 Compare November 5, 2026 10:59

@zhanghaohit zhanghaohit left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The keys match the continuation-gateway subchart (continuationModel, continuationEnabled, bufferToolCalls, stallIdleTimeoutSeconds), the form's defaults match the subchart's, and leaving continuationEngine to each parent chart is right. The minimum versions check out: vllm gained continuationGateway in 0.8.9, and sglang 0.8.11 follows #40 making continuationModel required.

Non-blocking notes:

  1. Prerelease charts are rejected. supportedChart only accepts a strict x.y.z, so 0.9.0-rc.1 counts as unsupported. Test clusters deploy rc charts routinely.
  2. The chart gate runs while recovery is off. recoveryError checks actual.configured && !supportedChart, so any layer that merely mentions continuationGateway triggers it, for example a catalog variant with enabled: false. Combined with (1), a deploy with recovery switched off fails with the chart error on an rc chart. I'd gate on enabled === true and compare versions with the prerelease suffix ignored (or semver-aware). An old chart ignores the unknown key anyway.
  3. The initial toggle state may not match the plan for a new deploy.
    • The form seeds from variant.values + profile.extra only, while the plan merges defaults / catalog / site / derived / form / edit / autoscale.
    • Suppose site or derived enables the gateway and profile.extra isn't that layer. The switch then shows off, matchesRecovery sees a mismatch, and the user gets overridden, which blames Advanced YAML for something it didn't do.
    • Can you confirm profile.extra is the site layer? Otherwise, consider seeding from a dry-run plan.
  4. Two small UX items.
    • An empty idle timeout shows nothing, while the effective value is the subchart's 600 s. A placeholder like "600 (chart default)" would help.
    • The tool-call buffering switch could be disabled while "Resume interrupted responses" is off, since the hint already says it has no effect then.

Expose existing continuationGateway chart settings in Model Serving deploy and upgrade forms, preserve recorded values, and validate the final plan before review.

Fixes modelsphere#63
@imneov
imneov force-pushed the feat/63-generation-recovery branch 2 times, most recently from 42ed37b to ffcbfd3 Compare November 6, 2026 13:56
Add regressions for higher-priority overrides, complete plan layers and timeout removal through validated recomposition. Preserve adaptive concurrency cleanup and reject unrelated changes between compositions.
@imneov
imneov merged commit bac3654 into modelsphere:main Nov 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

inferences: no way to turn on in-flight generation recovery (continuation_gateway) for a deployed model

2 participants