Repository navigation
Feat/mcp compass publish - #117
Open
operetz-rh wants to merge 10 commits into
Open
operetz-rh wants to merge 10 commits into
operetz-rh wants to merge 10 commits into
Conversation
operetz-rh
marked this pull request as draft
October 18, 2026 16:37
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
2 times, most recently
from
October 18, 2026 16:51
5f252e4 to
5fff2ec
Compare
operetz-rh
marked this pull request as ready for review
October 19, 2026 09:03
operetz-rh
marked this pull request as draft
October 26, 2026 09:56
Publish Phase 1+2 evaluation results to Compass SoundCheck on dedicated track. Includes CLI runner, bootstrap script, Tekton task, and tests. Staging validated against real MCP server and Compass staging.
Integrate Phase 3 behavioral evaluation results (from PR #116) into Compass SoundCheck publishing. Adds BehavioralEvalSummary model, Phase 3 check mappings (tool-correctness, edge-case-handling, sensitive-data- redaction), and Compass track with 3 behavioral checks. - Support behavioral-eval-summary.json parsing and fact publishing - Add Phase 3 to bootstrap_compass_track (fixes 400 error on track update) - Fix timestamp consistency across all fact payloads (single evaluated_at) - Fix Phase 3 naming via lookup table (consistency with Phase 1/2) - Fix Phase 3 optional payload fields (False/0 vs None for schema) Validated: 31 unit tests pass, real openshift-mcp-server eval published to Compass staging (HTTP 200), track visible with all phases. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
from
October 26, 2026 10:06
563b12a to
1f70384
Compare
Rename second TestSummaryPath to TestSummaryPathHelper to avoid F811 redefinition.
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
from
October 26, 2026 10:11
9fa28dc to
8b0f361
Compare
…lishing (dry-run + live modes). Live consume tool-name/oauth facts from API.
Refactors compass publish and bootstrap to support arbitrary phase counts, not just hardcoded phase1/2/3. Enables Phase 4+ by configuration only. Changes: - eval_result.py: Add phase3 to PHASE_CHECK_NAMES/SOURCE_DEFAULTS. Fix overall_score to default None pass scores to 1.0 not 0.0. - bootstrap_compass_track.py: Refactor to build checks/levels dynamically from PHASE_FACT_MAPS. Imports PHASE_FACT_MAPS instead of individual maps. - compass_publish.py: Already generic (uses .phases.items()). Fix doc reference phase2_compass_checks → conformance_eval_compass_checks. Fix evaluated_at to use passed param instead of hardcoding datetime.now(). - tests/test_mcp_compass_publish.py: Remove phase1_gate import/test, update bootstrap test for 3 levels. Tested live: 18 facts published to Compass staging (phase1/2/3 all present). All unit tests pass (36 compass + 76 phase tests). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Updated all phase name references from phase1/2/3 to new naming - Fixed runners to output correct phase names - Updated PHASE_FACT_MAPS, PHASE_SUMMARY_FACTS with new keys - Updated PHASE_CHECK_NAMES, PHASE_SOURCE_DEFAULTS - Updated bootstrap track metadata and order - Updated test fixtures and assertions - Verified e2e: Phase 1 real scans, Phase 2 live probes, publish to Compass - 36/36 tests pass, all code aligned Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…s-fresh Feat/mcp generic phases fresh
- Remove leftover conflict markers from rebase - Update example filenames to use new phase naming (static-eval, conformance-eval, behavioral-eval) - Include all 3 phases in example usage - Keep generic phase loader description (current state) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Update static_eval_gate.py docstring with correct phase name - Update run_mcp_eval.sh to reference static-eval-summary.json consistently - Fix variable reference from P1_REPORTS to REPORTS_DIR Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
operetz-rh
marked this pull request as ready for review
October 28, 2026 13:35
operetz-rh
force-pushed
the
feat/mcp-compass-publish
branch
from
October 28, 2026 14:25
79eef23 to
7ebc9fa
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the write side for MCP evaluation results — publish Phase 1+2 evaluation results to Compass SoundCheck on a dedicated track. Completes the reporting pipeline from local evaluation to centralized Compass dashboard for team visibility.
Changes
abevalflow/mcp/eval_result.py— backend-agnostic domain model (Phase 1/2/3) with summary loadersabevalflow/mcp/checks/compass_publish.py— Compass-specific publish logic with FactRef mappings and payload buildersscripts/mcp/runners/compass_publish.py— CLI runner with dry-run default (safety-first)scripts/mcp/bootstrap_compass_track.py— one-time track/check provisioning scriptpipeline/tasks/konflux/mcp-compass-publish.yaml— Tekton task for pipeline integrationtests/test_mcp_compass_publish.py— 20 unit tests covering models, loaders, payloads, publish flowscripts/mcp/README.md— documented staging flow, API shapes, commands, Phase 3 slottests/test_mcp_phase2.py— guards against raw-body SSE parsing bugTest plan
pytest tests/test_mcp_compass_publish.py— 20 tests passpytest tests/test_mcp_phase2.py— 46 tests pass (includes regression)openshift-mcp-server) against Compass staging:/api/soundcheck/facts/— all HTTP 200/api/soundcheck/results— HTTP 200https://compass.stage.redhat.com/soundcheck/tracks/abevalflow-mcp-evaluation/...