Repository navigation
feat: MCP behavioral evaluation with mcpchecker - #116
Merged
Merged
Conversation
Refactor Phase 1 (static) and Phase 2 (conformance) tasks to use Kubernetes Jobs for probe execution, enabling cluster-internal testing and better log visibility. **Phase 2 Job-based execution:** - New mcp-conformance-eval-job.yaml template for probe Job - Job runs phase2_conformance_probe and phase2_compass_probe - Single-line JSON markers (===REPORT_JSON:filename===) for reliable extraction - Raw log dump in run-probe-job step for debugging - Parse reports from logs instead of oc cp (works after pod completes) **Phase 1 improvements:** - Add parse-snapshot task integration - Update mcp-phase1-static for Job compatibility **RBAC refactoring:** - Scope SA permissions to ephemeral namespaces via RoleBinding - Remove ClusterRoleBinding (was granting cluster-wide access) - create-namespace task creates RoleBinding to activate mcp-eval-ephemeral-access - Remove runAsUser constraint to allow package installation in Jobs **Container image:** - Add Phase 2 Job template to mcp-eval image - Add envsubst for runtime config injection - Update comments to reflect Job-based architecture Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
IlonaShishov
force-pushed
the
phase3-behavioral-eval
branch
2 times, most recently
from
October 25, 2026 10:46
3913ed0 to
3950004
Compare
Add Phase 3 behavioral evaluation using mcpchecker for LLM agent-based testing of tool correctness, edge-case handling, and sensitive data redaction. Includes PR label gating and parallel phase execution. **Phase 3 Behavioral Evaluation:** - mcp-behavioral-eval task with Kubernetes Job execution - mcpchecker integration with configurable agent/judge LLM models - Default task suite: tool-correctness, edge-case-handling, sensitive-data-redaction - mcp-behavioral-eval-job.yaml template with LLM env vars - Raw log dump for debugging mcpchecker runs (matches Phase 2 pattern) - Report aggregation with configurable pass threshold (default 0.7) - 1800s Job timeout (30 minutes for LLM-based evaluation) **PR Label Gating:** - check-label task fetches PR labels from GitHub API - External task design for cross-cluster PR metadata access - Gate Phase 3 execution on "mcpchecker" label presence - Added to integration pipeline with conditional execution **Pipeline Integration:** - Add Phase 3 to konflux-mcp-eval-pipelinerun.yaml - Parallel execution of Phase 1, Phase 2, and Phase 3 - Update Makefile with check-label and mcp-behavioral-eval tasks **mcpchecker Suite:** - suites/mcp/default/eval.yaml: suite configuration - suites/mcp/default/mcp-config.yaml: MCP server connection config - suites/mcp/default/tasks/*.yaml: task definitions - Baked into mcp-eval container image **Container Updates:** - mcpchecker-runner multi-stage build for binary/Node.js - Copy mcpchecker suite files to /opt/suites/mcp/default - Add aggregate_mcpchecker.py for result processing Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
IlonaShishov
force-pushed
the
phase3-behavioral-eval
branch
2 times, most recently
from
October 25, 2026 14:15
7178e27 to
6a8bdf8
Compare
- Separate summaries and detailed results into reports/ and results/ - Rename phase-based to evaluation-type naming convention - Fix submission-name semantics and flatten directory structure - Remove debug print steps and simplify log collection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IlonaShishov
force-pushed
the
phase3-behavioral-eval
branch
from
October 25, 2026 14:17
6a8bdf8 to
e93d62d
Compare
operetz-rh
added a commit
that referenced
this pull request
Oct 26, 2026
Integrate Phase 3 behavioral evaluation results (from PR #116) into Compass SoundCheck publishing. Adds BehavioralEvalSummary model, Phase 3 check mappings (tool-correctness, edge-case-handling, sensitive-data- redaction), and Compass track with 3 behavioral checks. - Support behavioral-eval-summary.json parsing and fact publishing - Add Phase 3 to bootstrap_compass_track (fixes 400 error on track update) - Fix timestamp consistency across all fact payloads (single evaluated_at) - Fix Phase 3 naming via lookup table (consistency with Phase 1/2) - Fix Phase 3 optional payload fields (False/0 vs None for schema) Validated: 31 unit tests pass, real openshift-mcp-server eval published to Compass staging (HTTP 200), track visible with all phases. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
operetz-rh
added a commit
that referenced
this pull request
Oct 26, 2026
Integrate Phase 3 behavioral evaluation results (from PR #116) into Compass SoundCheck publishing. Adds BehavioralEvalSummary model, Phase 3 check mappings (tool-correctness, edge-case-handling, sensitive-data- redaction), and Compass track with 3 behavioral checks. - Support behavioral-eval-summary.json parsing and fact publishing - Add Phase 3 to bootstrap_compass_track (fixes 400 error on track update) - Fix timestamp consistency across all fact payloads (single evaluated_at) - Fix Phase 3 naming via lookup table (consistency with Phase 1/2) - Fix Phase 3 optional payload fields (False/0 vs None for schema) Validated: 31 unit tests pass, real openshift-mcp-server eval published to Compass staging (HTTP 200), track visible with all phases. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces Phase 3 behavioral evaluation capabilities through mcpchecker integration and refactors the existing evaluation phases for improved clarity and execution model.
Key changes:
mcp-behavioral-evalTekton Task that deploys a mcpchecker Job on the target cluster, collects results, and aggregates them viaaggregate_mcpchecker.pystatic-eval)conformance-eval)behavioral-eval)suites/mcp/default/with tool-correctness, edge-case-handling, and sensitive-data-redaction testsMCPCHECKER_LABELparameter)parse-snapshottask to expose genericpr-labelspassthrough resultAssociated Jira: APPENG-6673
🤖 Generated with Claude Code