Skip to content

fix: extract LangChain content-block text and apply model parameters after eval - #80

Merged
andrewklatzke merged 1 commit into
mainfrom
aklatzke/AIC-3351/fix-content-blocks
Sep 16, 2026
Merged

andrewklatzke merged 1 commit into
mainfrom
aklatzke/AIC-3351/fix-content-blocks

Conversation

@andrewklatzke

Copy link
Copy Markdown
Contributor

Summary

  • Extract visible text from LangChain AIMessage.content when it is a list of thinking/text blocks so callers no longer get an empty output while tokens are still billed.
  • Accept llm as a factory (config) => model (in addition to a pre-built instance) and spread flag model.parameters unchanged into default ChatAnthropic/ChatOpenAI constructors after evaluation.
  • Add langchain-thinking example (tool-free prompt) plus unit tests for mixed content blocks, factory vs instance, and default constructor kwargs.

Test plan

  • packages/langchain-messages and packages/langchain-agents unit tests
  • python main.py langchain-thinking launch-darkly-documentation-summarizer-messages-claude "Reason it out yourself without any tools: what is 17 times 23?" — non-empty response
  • Same scenario with doesnt-exist — exit 1

Made with Cursor

…after eval

LangChain AIMessage.content can be a list of thinking/text blocks; keep the visible text for callers. Accept llm as a factory so ChatAnthropic/ChatOpenAI can be built with flag model.parameters after evaluation.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7beaa6f. Configure here.

model_name=str(model.get("name") or "claude-sonnet-4-5"),
thinking={"type": "enabled", "budget_tokens": _THINKING_BUDGET_TOKENS},
max_tokens_to_sample=_MAX_TOKENS,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example factory kwargs collide with parameters

Medium Severity

The thinking example spreads flag model.parameters into ChatAnthropic and then passes overlapping keywords (timeout, stop, model_name, thinking, max_tokens_to_sample). A second value for any of those keys is a TypeError, and max_tokens aliases max_tokens_to_sample, so a typical Claude flag never constructs the model.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 7beaa6f. Configure here.



def _make_ai_msg(
content: str = "answer", input_tokens: int = 10, output_tokens: int = 5

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious - why the change from string -> any?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is specifically targeting langchain's "content blocks" pattern it uses for some providers. So no longer a string. Any type we provided here given the breadth of what it can return would be incorrect for someone depending on provider, etc.

@andrewklatzke
andrewklatzke merged commit a4e1aad into main Sep 16, 2026
8 checks passed
@andrewklatzke
andrewklatzke deleted the aklatzke/AIC-3351/fix-content-blocks branch September 16, 2026 23:04
@github-actions github-actions Bot mentioned this pull request Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants