Skip to content

Generic phase loaders support - #118

Closed
operetz-rh wants to merge 3 commits into
feat/mcp-compass-publishfrom
feat/mcp-generic-phases
Closed

operetz-rh wants to merge 3 commits into
feat/mcp-compass-publishfrom
feat/mcp-generic-phases

Conversation

@operetz-rh

Copy link
Copy Markdown

Summary

Implements the write side for MCP evaluation results — generalize Phase 1+2 evaluation logic to support arbitrary phases, then publish results to Compass SoundCheck. Completes the reporting pipeline from local evaluation to centralized Compass dashboard for team visibility.

Changes

  • abevalflow/mcp/eval_result.py — generic domain model supporting Phase 1/2/3+ with pluggable loaders
  • abevalflow/mcp/checks/compass_publish.py — Compass-specific publish logic with FactRef mappings and payload builders
  • scripts/mcp/runners/compass_publish.py — CLI runner with dry-run default (safety-first)
  • scripts/mcp/bootstrap_compass_track.py — one-time track/check provisioning script
  • pipeline/tasks/konflux/mcp-compass-publish.yaml — Tekton task for pipeline integration
  • tests/test_mcp_compass_publish.py — 35 unit tests covering generic loaders, payloads, publish flow
  • scripts/mcp/README.md — documented local/cluster flows, Compass integration, entry points

Test plan

  • pytest tests/test_mcp_compass_publish.py — 35 tests pass
  • pytest tests/test_mcp_phase1.py tests/test_mcp_phase2.py — 108 tests pass
  • Live validation on mock MCP server:
    • Phase 1 FAIL (0.75 grade) / Phase 2 PASS (1.0 grade, 4/10 checks)
    • Dry-run: 10 facts logged (no HTTP)
    • Live publish: all 10 facts → Compass (HTTP 200)
  • Generic phase support verified:
    • Loader accepts Phase 4, Phase 1a, Phase 1b (no hardcoding)
    • FactRef mappings warn on unknown phases, proceed safely

Publish Phase 1+2 evaluation results to Compass SoundCheck on dedicated track.
Includes CLI runner, bootstrap script, Tekton task, and tests.
Staging validated against real MCP server and Compass staging.
@operetz-rh operetz-rh self-assigned this Oct 22, 2026
…lishing

(dry-run + live modes). Live consume tool-name/oauth facts from API.
@operetz-rh
operetz-rh force-pushed the feat/mcp-generic-phases branch from aff6c67 to 6bdf460 Compare October 22, 2026 09:43
@operetz-rh
operetz-rh force-pushed the feat/mcp-compass-publish branch from 563b12a to 1f70384 Compare October 26, 2026 10:06
@operetz-rh operetz-rh closed this Oct 26, 2026
@operetz-rh

Copy link
Copy Markdown
Author

#122

@operetz-rh operetz-rh reopened this Oct 28, 2026
@operetz-rh operetz-rh closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant