feat: MCP behavioral evaluation with mcpchecker

abevalflow/mcp/README.md

@@ -1,11 +1,11 @@
 # `abevalflow/mcp/` — MCP evaluation pipeline logic
 
-All evaluation logic for the MCP-server evaluation pipeline (ADR Approach 4,
-Phases 1 and 2). This package **reads** result JSON, applies policy, and
-**decides** pass/fail. No shell-outs to external tools, no network calls.
-Testable with JSON fixtures.
+All evaluation logic for the MCP-server evaluation pipeline (ADR Approach 4:
+static, conformance, and behavioral evaluations). This package **reads** result
+JSON, applies policy, and **decides** pass/fail. No shell-outs to external tools,
+no network calls. Testable with JSON fixtures.
 
-## Structure (organized by role, phase-prefixed)
+## Structure (organized by role and evaluation type)
 
 | Directory | Role | What it does |
 |-----------|------|-------------|
@@ -14,20 +14,20 @@ Testable with JSON fixtures.
 
 ### `gates/` — all pass/fail gates
 
-| File | Phase | What it does |
-|------|-------|-------------|
-| `phase1_secrets_gate.py` | 1 | `SecretsGate` — reads `secrets-scan.json`, counts HIGH findings |
-| `phase1_no_user_code_gate.py` | 1 | `NoUserCodeGate` — reads `no-user-code-scan.json`, counts HIGH findings |
-| `phase1_license_gate.py` | 1 | `LicenseGate` — reads `license-scan.json`, checks against allow-list |
-| `phase2_conformance_gate.py` | 2 | `Phase2Gate` — reads any `<check>-check.json`, maps three-state to `GateResult` |
-| `__init__.py` | 1+2 | Orchestrators: `run_phase1()`, `run_phase2()`, gate lists |
+| File | Evaluation Type | What it does |
+|------|-----------------|-------------|
+| `static_eval_secrets_gate.py` | Static | `SecretsGate` — reads `secrets-scan.json`, counts HIGH findings |
+| `static_eval_no_user_code_gate.py` | Static | `NoUserCodeGate` — reads `no-user-code-scan.json`, counts HIGH findings |
+| `static_eval_license_gate.py` | Static | `LicenseGate` — reads `license-scan.json`, checks against allow-list |
+| `conformance_eval_gate.py` | Conformance | `ConformanceEvalGate` — reads any `<check>-check.json`, maps three-state to `GateResult` |
+| `__init__.py` | All | Orchestrators: `run_static_eval()`, `run_conformance_eval()`, gate lists |
 
 ### `checks/` — conformance probes and Compass-consumed checks
 
-| File | Phase | What it does |
-|------|-------|-------------|
-| `phase2_conformance_checks.py` | 2 | 8 `check_*` functions + `probe_all()` — conformance checks against a running MCP server |
-| `phase2_compass_checks.py` | 2 | `map_facts()` + `map_results()` — maps Compass facts/results to `CheckOutcome` |
+| File | Evaluation Type | What it does |
+|------|-----------------|-------------|
+| `conformance_eval_checks.py` | Conformance | 8 `check_*` functions + `probe_all()` — conformance checks against a running MCP server |
+| `conformance_eval_compass_checks.py` | Conformance | `map_facts()` + `map_results()` — maps Compass facts/results to `CheckOutcome` |
 
 ### Root modules
 
@@ -40,31 +40,31 @@ Testable with JSON fixtures.
 ## Data flow
 
 ```
-Phase 1 (static — no running server)
+Static Evaluation (build-time — no running server)
 ┌─────────────────────────────────────────────────────────────────┐
 │ scripts/mcp/scanners/                                           │
-│   phase1_secrets_scan       ──→ secrets-scan.json                │
-│   phase1_no_user_code_scan  ──→ no-user-code-scan.json           │
-│   phase1_license_scan       ──→ license-scan.json                │
+│   static_eval_secrets_scan       ──→ secrets-scan.json          │
+│   static_eval_no_user_code_scan  ──→ no-user-code-scan.json     │
+│   static_eval_license_scan       ──→ license-scan.json          │
 └──────────────────────────────────┬──────────────────────────────┘
                                    │ JSON on disk
 ┌──────────────────────────────────▼──────────────────────────────┐
 │ abevalflow/mcp/gates/                                           │
-│   Phase1SecretsGate     ──→ GateResult (pass/fail)               │
-│   Phase1NoUserCodeGate  ──→ GateResult (pass/fail)               │
-│   Phase1LicenseGate     ──→ GateResult (pass/fail)               │
+│   SecretsGate           ──→ GateResult (pass/fail)              │
+│   NoUserCodeGate        ──→ GateResult (pass/fail)              │
+│   LicenseGate           ──→ GateResult (pass/fail)              │
 └─────────────────────────────────────────────────────────────────┘
 
-Phase 2 (live — against the running MCP server)
+Conformance Evaluation (live — against the running MCP server)
 ┌─────────────────────────────────────────────────────────────────┐
 │ scripts/mcp/probes/                                             │
-│   phase2_conformance_probe   ──→ <check>-check.json (per check)       │
-│   phase2_compass_probe ──→ <check>-check.json (consumed facts)  │
+│   conformance_eval_probe   ──→ <check>-check.json (per check)  │
+│   conformance_eval_compass_probe ──→ <check>-check.json (facts)│
 └──────────────────────────────────┬──────────────────────────────┘
                                    │ JSON on disk
 ┌──────────────────────────────────▼──────────────────────────────┐
 │ abevalflow/mcp/gates/                                           │
-│   Phase2ConformanceGate (×N)  ──→ GateResult (pass/fail/not_evaluated) │
+│   ConformanceEvalGate (×N)  ──→ GateResult (pass/fail/not_eval) │
 └─────────────────────────────────────────────────────────────────┘
 ```
 
@@ -74,9 +74,9 @@ This package contains the **what** (evaluation logic — reads JSON, decides).
 `scripts/mcp/` contains the **how** (CLI entry points — runs tools, writes JSON).
 The contract between them is JSON files on disk.
 
-## Three-state model (Phase 2)
+## Three-state model (Conformance Evaluation)
 
-Every Phase 2 check produces one of three states:
+Every conformance evaluation check produces one of three states:
 
 | State | Meaning | Effect on verdict | Effect on grade |
 |---|---|---|---|
@@ -86,17 +86,17 @@ Every Phase 2 check produces one of three states:
 
 A server is never penalized for something it was never tested for.
 
-## Adding a new check
+## Adding a new conformance check
 
-1. **Add the check function** in `checks/phase2_conformance_checks.py` — takes the
+1. **Add the check function** in `checks/conformance_eval_checks.py` — takes the
    client and/or a response, returns a `CheckOutcome`.
 2. **Wire it** in `probe_all()` (same file).
-3. **Add the check name** to `PHASE2_CHECKS` in `gates/__init__.py`.
+3. **Add the check name** to `CONFORMANCE_EVAL_CHECKS` in `gates/__init__.py`.
 4. The gate picks it up automatically — it reads any `<check>-check.json` file.
    No gate code changes needed.
 
 ## Testing
 
 All code in this package is testable with JSON fixtures — no running server,
-no external tools. Tests are in `tests/test_mcp_phase1.py` and
-`tests/test_mcp_phase2.py`.
+no external tools. Tests are in `tests/test_mcp_static_eval.py` and
+`tests/test_mcp_conformance_eval.py`.

abevalflow/mcp/__init__.py

@@ -5,17 +5,17 @@
 Isolated from the skill evaluation pipeline (see ADR: Evaluation Strategy for
 MCP Servers as Standalone Software Components, Approach 4).
 
-Layout (organized by role, phase-prefixed):
+Layout (organized by role and evaluation type):
 
 - ``types``       - core types: ``CheckOutcome``, ``STATUS_*``
 - ``helpers``     - finding builders, severity normalization, check-result I/O
 - ``mcp_client``  - MCP protocol client (JSON-RPC over Streamable HTTP)
-- ``gates/``      - all pass/fail gates (Phase 1 + Phase 2)
-- ``checks/``     - conformance check functions + Compass mapping (Phase 2)
+- ``gates/``      - all pass/fail gates (static + conformance evaluation)
+- ``checks/``     - conformance check functions + Compass mapping
 
 The CLI entry points that invoke this logic and handle I/O (shell out to
 scanners, create an MCP client, read/write files) live in ``scripts/mcp/``.
 
-Phases 1 and 2 are implemented here; Phase 3 (behavioral, mcpchecker) is out of
-scope for now.
+Static and conformance evaluations are implemented here; behavioral evaluation
+(mcpchecker) is out of scope for now.
 """

abevalflow/mcp/checks/phase2_conformance_checks.py → abevalflow/mcp/checks/conformance_eval_checks.py

@@ -1,12 +1,12 @@
-"""Phase 2 live conformance check functions.
+"""Conformance evaluation live check functions.
 
 Each check takes a ``MCPClient`` (and/or the ``tools/list`` response) and returns
 a ``CheckOutcome`` carrying a three-state result (pass / fail / not_evaluated).
 ``probe_all`` orchestrates every check against a running server.
 
 These are pure evaluation logic: the client handles I/O, and the checks analyze
 responses and return verdicts. The CLI entry point that creates the client and
-writes results to disk lives in ``scripts/mcp/probes/phase2_conformance_probe.py``.
+writes results to disk lives in ``scripts/mcp/probes/conformance_eval_probe.py``.
 """
 
 from __future__ import annotations
@@ -223,7 +223,7 @@ def check_schema_conformance(client: MCPClient, tools_resp: RpcResponse) -> Chec
     inputSchema is required by the MCP tool type, so a missing/non-object one is a
     fail. outputSchema is optional in the MCP spec, so it is validated only when
     present (its absence is not a violation). Whether actual tool RESPONSES conform
-    to the declared schema is behavioral and is left to Phase 3.
+    to the declared schema is behavioral and is left to behavioral evaluation.
     """
     err = _tools_list_error(tools_resp)
     if err:
@@ -283,7 +283,7 @@ def check_tool_annotations(client: MCPClient, tools_resp: RpcResponse) -> CheckO
     conformance violation: a server that declares none is not_evaluated here
     (nothing to verify), never failed. Only a *malformed* annotation (a hint
     field present with a non-boolean value) is a real fail. Honesty of the hints
-    is behavioral and is left to Phase 3.
+    is behavioral and is left to behavioral evaluation.
     """
     err = _tools_list_error(tools_resp)
     if err:
@@ -524,7 +524,7 @@ def check_openapi_conformance(
 
 
 def probe_all(server_url: str, *, timeout: float = 30.0, openapi_url: str | None = None) -> list[CheckOutcome]:
-    """Run every live Phase 2 check against ``server_url``.
+    """Run every live conformance evaluation check against ``server_url``.
 
     If the server is unreachable, all checks are reported not_evaluated (ADR:
     a readiness/reachability failure must never be scored as a fail).

abevalflow/mcp/checks/phase2_compass_checks.py → abevalflow/mcp/checks/conformance_eval_compass_checks.py

@@ -1,9 +1,9 @@
-"""Phase 2 Compass-consumed check mapping.
+"""Conformance evaluation Compass-consumed check mapping.
 
-A few Phase 2 checks are already collected by Compass as real automated facts, so
-the pipeline consumes them rather than re-probing (avoids duplicating what
-Compass does well). This module maps those Compass SoundCheck facts into the same
-three-state ``CheckOutcome`` the Phase 2 gates read.
+A few conformance evaluation checks are already collected by Compass as real
+automated facts, so the pipeline consumes them rather than re-probing (avoids
+duplicating what Compass does well). This module maps those Compass SoundCheck
+facts into the same three-state ``CheckOutcome`` the conformance evaluation gates read.
 
 Consumed checks and their source facts::
 
@@ -26,7 +26,7 @@
   results from ``GET /api/soundcheck/results``.
 
 The CLI entry point that reads facts from a file or fetches live results from
-the SoundCheck API lives in ``scripts/mcp/probes/phase2_compass_probe.py``.
+the SoundCheck API lives in ``scripts/mcp/probes/conformance_eval_compass_probe.py``.
 """
 
 from __future__ import annotations
@@ -48,8 +48,8 @@
 CHECK_TOOL_NAME = "mcp-tool-schema-check"
 CHECK_OAUTH = "mcp-oauth-scope-match"
 
-# Maps a SoundCheck check ID to the Phase 2 check name our gates read.
-_CHECK_ID_TO_PHASE2 = {
+# Maps a SoundCheck check ID to the conformance evaluation check name our gates read.
+_CHECK_ID_TO_CONFORMANCE_EVAL = {
     CHECK_TOOL_NAME: "tool-name-rules",
     CHECK_OAUTH: "oauth-catalog-match",
 }
@@ -76,7 +76,7 @@ def _bool_outcome(check: str, value: Any, ok_reason: str, bad_reason: str) -> Ch
 
 
 def map_facts(facts: dict[str, Any]) -> list[CheckOutcome]:
-    """Map Compass facts to the consumed Phase 2 check outcomes."""
+    """Map Compass facts to the consumed conformance evaluation check outcomes."""
     outcomes: list[CheckOutcome] = []
 
     outcomes.append(
@@ -159,7 +159,7 @@ def map_facts(facts: dict[str, Any]) -> list[CheckOutcome]:
 
 
 def map_results(results: list[dict]) -> list[CheckOutcome]:
-    """Map SoundCheck API check results to Phase 2 check outcomes.
+    """Map SoundCheck API check results to conformance evaluation check outcomes.
 
     ``results`` is the ``results`` array from
     ``GET /api/soundcheck/results?entityRef=...&checks=...``.  Each element has
@@ -171,12 +171,12 @@ def map_results(results: list[dict]) -> list[CheckOutcome]:
     by_check_id = {r["checkId"]: r for r in results if isinstance(r, dict) and "checkId" in r}
     outcomes: list[CheckOutcome] = []
 
-    for check_id, phase2_name in _CHECK_ID_TO_PHASE2.items():
+    for check_id, check_name in _CHECK_ID_TO_CONFORMANCE_EVAL.items():
         result = by_check_id.get(check_id)
         if result is None:
             outcomes.append(
                 CheckOutcome(
-                    phase2_name,
+                    check_name,
                     STATUS_NOT_EVALUATED,
                     f"No Compass result for {check_id}; entity may not be registered yet.",
                     source="compass",
@@ -196,25 +196,23 @@ def map_results(results: list[dict]) -> list[CheckOutcome]:
 
         if state == "passed":
             outcomes.append(
-                CheckOutcome(phase2_name, STATUS_PASS, notes or f"Compass check {check_id} passed.", source="compass")
+                CheckOutcome(check_name, STATUS_PASS, notes or f"Compass check {check_id} passed.", source="compass")
             )
         elif state == "failed":
             outcomes.append(
                 CheckOutcome(
-                    phase2_name,
+                    check_name,
                     STATUS_FAIL,
                     notes or f"Compass check {check_id} failed.",
                     source="compass",
-                    findings=[
-                        make_finding(severity="high", message=notes or f"{check_id} failed", rule_id=phase2_name)
-                    ],
+                    findings=[make_finding(severity="high", message=notes or f"{check_id} failed", rule_id=check_name)],
                 )
             )
         else:
             # "error", "not-applicable", or unknown state
             outcomes.append(
                 CheckOutcome(
-                    phase2_name,
+                    check_name,
                     STATUS_NOT_EVALUATED,
                     f"Compass check {check_id} returned state '{state}': {notes}".strip(),
                     source="compass",