Skip to content

DO NOT MERGE: native eval CI proof for TC-7141 - #354

Closed
mrizzi wants to merge 1 commit into
RHEcosystemAppEng:mainfrom
mrizzi:tc7141-native-eval-proof
Closed

mrizzi wants to merge 1 commit into
RHEcosystemAppEng:mainfrom
mrizzi:tc7141-native-eval-proof

Conversation

@mrizzi

@mrizzi mrizzi commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Do not merge. This draft PR only triggers the Native Fullsend Evals on the stack merged in #353: fullsend v0.47.1 (c68903c0), OpenShell 0.1.2, the built-in vertex-ai provider. That covers TC-7141's remaining acceptance criterion. The only change is an HTML comment in a triage-security reference file. The PR will be closed without merging once the runs finish.

Refs TC-7141

🤖 Generated with Claude Code

Summary by Sourcery

Run Native Fullsend Evals to complete the remaining TC-7141 acceptance evidence without introducing functional product changes.

CI:

  • Trigger the Native Fullsend Evals needed to provide CI evidence for TC-7141 on the specified fullsend and OpenShell stack.

Chores:

  • Mark the draft PR as temporary CI proof and not intended for merging.

Touches a triage-security skill file only to make eval-pr-run.yml run the
native Fullsend suite on fullsend v0.47.1 / OpenShell 0.1.2. The draft PR
is closed without merging.

Refs TC-7141

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Code
@sourcery-ai

sourcery-ai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor
Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

This draft, explicitly not intended for merging, changes only a reference Markdown file to trigger Native Fullsend Evals against fullsend v0.47.1, OpenShell 0.1.2, and the built-in vertex-ai provider; it should be closed after the CI runs complete.

File-Level Changes

Change Details Files
Add a non-functional marker to trigger Native Fullsend Evals for the specified dependency stack and document the CI proof context.
  • Append an HTML comment identifying TC-7141 and the evaluated fullsend/OpenShell versions.
  • Leave runtime behavior and skill instructions unchanged.
plugins/sdlc-workflow/skills/triage-security/version-impact-analysis.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@github-actions

Copy link
Copy Markdown
Contributor

Eval Results

Source head: f921eb2
Merge: 9f5d439

Eval Results: triage-security

Eval Passed Failed Pass Rate
eval-1 13/13 0 100%
eval-10 5/5 0 100%
eval-11 5/5 0 100%
eval-12 5/5 0 100%
eval-13 5/5 0 100%
eval-14 5/5 0 100%
eval-15 5/5 0 100%
eval-16 7/7 0 100%
eval-17 5/5 0 100%
eval-18 5/5 0 100%
eval-19 5/5 0 100%
eval-2 5/5 0 100%
eval-20 4/4 0 100%
eval-21 4/4 0 100%
eval-22 4/4 0 100%
eval-23 4/4 0 100%
eval-24 4/4 0 100%
eval-25 4/4 0 100%
eval-26 5/5 0 100%
eval-27 5/5 0 100%
eval-28 5/5 0 100%
eval-29 5/5 0 100%
eval-3 5/5 0 100%
eval-30 4/4 0 100%
eval-31 4/4 0 100%
eval-32 4/4 0 100%
eval-4 5/5 0 100%
eval-5 6/6 0 100%
eval-6 6/6 0 100%
eval-7 5/5 0 100%
eval-8 8/8 0 100%
eval-9 5/5 0 100%

Pass rate: 100% · Tokens: 3,951 · Duration: 5s

Baseline (f32bd713): 99% · 59,221 tokens · 139s


Generated by sdlc-workflow/run-evals v0.13.10

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Native Fullsend Eval Results

Head: f921eb2
Merge: 9f5d439
Trusted workflow: e1c811c
Reviewed eval source: e1c811c

Native job: success; complete: true.
Execution evidence: valid; quality score (advisory): 27/28 passed.

Case Execution evidence Quality score (advisory)
033-absent valid 4/4
034-empty valid 5/5
035-malformed valid 4/5
036-valid valid 7/7
037-release valid 7/7

@mrizzi

mrizzi commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

Native eval CI proof for TC-7141 complete; closing without merging. Eval PR Run 39989276781: Native Fullsend Evals ran on OpenShell 0.1.2 / fullsend v0.47.1 with eval source e1c811c. execution_valid=true, complete, 27/28 (035-malformed assertion_1 false, which is advisory and matches the pre-change results on #349). Ordinary triage-security evals: 100%.

@mrizzi mrizzi closed this Oct 23, 2026
@mrizzi
mrizzi deleted the tc7141-native-eval-proof branch October 23, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant