Skip to content
This repository was archived by the owner on Nov 15, 2025. It is now read-only.
This repository was archived by the owner on Nov 15, 2025. It is now read-only.

[Feature] Optimize Prompts for Direct LLM Calls in Non-Agentic Workflows #16

Description

@christso

Description:

Problem: Users need to optimize system prompts for simple, direct LLM calls (e.g., via OpenAI API) for tasks like SQL code reviews or schema change validation, incorporating company-specific guidelines (e.g., attached PDF/YAML files for style rules).

Proposed Solution: Extend bbeval opt to support direct LLM optimization:

  • Input: Base system prompt + attached guidelines file (e.g., --guidelines coding_rules.pdf).
  • Use: BootstrapRS (≤5 trials) to tune prompt for accuracy on test tasks (e.g., "Review this SQL: [code]" → scored against expected feedback).
  • Output: JSON prompt with enforced rules (e.g., required placeholders like {{guidelines}}, banned phrases).
  • Integration: --mode direct-llm flag; validate via code_execution tool for SQL syntax.

Benefits: Enables quick tuning for non-agentic reviews; reduces manual prompt engineering by 80% on guideline adherence.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions