Skip to content

fix(server): accept max_completion_tokens in chat requests - #2386

Open
laulauraluo wants to merge 2 commits into
abetlen:mainfrom
laulauraluo:fix/chat-completion-token-limit
Open

laulauraluo wants to merge 2 commits into
abetlen:mainfrom
laulauraluo:fix/chat-completion-token-limit

Conversation

@laulauraluo

Copy link
Copy Markdown

Chat requests using max_completion_tokens currently pass max_tokens=None to the backend, dropping the requested limit. Accept the new name as a validation alias while keeping the existing backend keyword, defaults and max_tokens precedence when both names are supplied.

Fixes #1907.

Validation:

  • 15 model-free tests passed, including the new request-validation regressions.
  • Six regular/streaming HTTP cases passed through the actual FastAPI router with a mocked model backend.
  • make lint passed.

Validated on Python 3.12 with a CPU-only native build. No model inference or full model-dependent test suite was run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openai API max_completion_tokens argument is ignored

1 participant