Skip to content

Improve plan-feature shared-artifact coverage - #359

Merged
mrizzi merged 4 commits into
RHEcosystemAppEng:mainfrom
ruromero:TC-7146
Oct 28, 2026
Merged

mrizzi merged 4 commits into
RHEcosystemAppEng:mainfrom
ruromero:TC-7146

Conversation

@ruromero

@ruromero ruromero commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

plan-feature/
├── Step 3: trace shared artifacts through all writers, consumers, fallbacks, and on-demand paths
├── Step 5: carry paths, validations, tests, and eval coverage into tasks; surface scope conflicts
└── evals: add multi-writer coverage and scope-conflict cases
docs/constraints.md: add constraint 1.106
CLAUDE.md: add approved Jira defaults and generic Serena tool naming

Implements TC-7146.

Evidence

  • No-guidance baseline omitted the concrete alternate writer/test paths; final eval 7 passes 10/10 and eval 8 4/4.
  • uv run --with pytest --with pyyaml --with 'jsonschema[format]' --no-project -- python -m pytest plugins/sdlc-workflow/scripts/ -q — 740 passed, 3 skipped (temporary environment; no repo dependency changes).
  • uvx skillsaw — no error-level findings (warnings only).
  • claude plugin validate plugins/sdlc-workflow — passed with an icon warning.
  • Full plan-feature suite: eval 4 10/11, eval 6 12/14. Remaining assertions concern convention applicability and bookend sections; these were not baseline-confirmed.
  • evals/plan-feature/README.md still lists cases 1–4; left unchanged because it is outside TC-7146's approved file scope.

Merge Danger

Door: Two-way; the guidance and eval additions are revertible.

Blast Radius: Low; affects planning outputs and project workflow documentation.

Summary by Sourcery

Ensure plan-feature plans account for complete shared-artifact lifecycle coverage and expose scope conflicts before claiming acceptance criteria are satisfied.

New Features:

  • Expand plan-feature guidance to trace shared artifacts across all producer, consumer, fallback, recovery, and on-demand paths.
  • Add evaluation scenarios covering multiple shared-artifact writers and conflicting scope constraints.

Enhancements:

  • Require generated implementation tasks to carry affected paths, path-specific validation, tests, and unresolved scope conflicts for shared-artifact criteria.
  • Document Jira field defaults and generic Serena tool naming conventions.
  • Add a constraint governing complete shared-artifact lifecycle coverage in feature planning.

Documentation:

  • Update workflow documentation with Jira defaults, Serena naming guidance, and shared-artifact planning requirements.

Tests:

  • Extend plan-feature evaluation coverage for multi-writer shared artifacts and scope-conflict handling.

@sourcery-ai

sourcery-ai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

This PR updates plan-feature to analyze shared artifacts across every producer, consumer, fallback, and on-demand path, then carry concrete paths, validation, tests, and eval coverage into tasks while surfacing scope conflicts. It adds targeted multi-writer evaluation fixtures and records the new requirement as constraint 1.106, alongside workflow documentation for Jira defaults and Serena naming.

Flow diagram for shared-artifact coverage in plan-feature

flowchart LR
    A["Feature acceptance criterion"] --> B["Step 3: map shared artifact"]
    B --> C["Trace all producers and writers"]
    B --> D["Trace consumers and presentations"]
    B --> E["Trace fallback and on-demand paths"]
    C --> F["Step 5: reconcile into tasks"]
    D --> F
    E --> F
    F --> G["Files, validation, tests, and eval coverage"]
    F --> H["Surface scope conflicts"]
Loading

File-Level Changes

Change Details Files
Strengthen plan-feature guidance to trace shared artifacts across their complete lifecycle and propagate that analysis into implementation tasks.
  • Require discovery of all primary, scheduled, fallback, recovery, on-demand, and API writers.
  • Require identification of every consumer plus associated validation and test paths.
  • Require concrete source and test paths in the impact map.
  • Require unresolved scope conflicts when excluded paths are needed for acceptance criteria.
  • Require task files, validation, tests, and applicable eval coverage to reflect every affected path.
  • Preserve existing behavior such as stable identifiers and update semantics in test requirements.
plugins/sdlc-workflow/skills/plan-feature/SKILL.md
docs/constraints.md
Expand plan-feature evaluation coverage with synthetic shared-artifact scenarios that test multi-writer tracing and scope-conflict handling.
  • Add a fictional shared Service Catalog feature with registration, scheduled sync, fallback import, and on-demand refresh writers.
  • Cover summary API and downloadable-report consumers.
  • Add requirements for path-specific tests, documentation impact, and preservation of existing behavior.
  • Add multi-writer and conflicting-scope evaluation cases.
evals/plan-feature/evals.json
evals/plan-feature/files/feature-shared-artifact-multi-writer.md
Document Jira defaults and generic Serena tool naming conventions for workflow use.
  • Document default priority, fixVersion scope, and prompting behavior.
  • Specify instance-prefixed Serena MCP tool names with an example.
CLAUDE.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Needs a human reviewer. If the new planning guidance overstates which producer or consumer paths are affected, it could lead to unnecessary or incomplete Jira tasks being created. Those issues remain after a revert, but they are bounded and can be corrected or closed; reverting stops future plans from using the guidance.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

@ruromero ruromero added the ok-to-test Enable verify-pr execution on PRs from forks label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Eval Results

Source head: 5270a81
Merge: a9b7e27

Eval Results: plan-feature

Eval Passed Failed Pass Rate
eval-1 19/19 0 100%
eval-2 16/16 0 100%
eval-3 15/15 0 100%
eval-4 11/11 0 100%
eval-5 14/15 1 93%
eval-6 14/14 0 100%
eval-7 10/10 0 100%
eval-8 4/4 0 100%

Failed Assertions

eval-5: 1 failing assertion
  • Assertion: "Each non-documentation task file contains all required template sections: Repository, Target Branch, Description, at least one of Files to Modify or Files to Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation tasks are exempt from requiring Files to Modify, Files to Create, and Implementation Notes — they must still include Repository, Target Branch, Description, Acceptance Criteria, and Test Requirements"
    Evidence: "Implementation tasks 2-7 all contain every required section (Repository, Target Branch, Description, Files to Modify/Create, Implementation Notes, Acceptance Criteria, Test Requirements). Documentation task 8 correctly includes Repository, Target Branch, Description, Acceptance Criteria, and Test Requirements (exempt from Files to Modify/Create and Implementation Notes). However, bookend tasks 1 and 9 are non-documentation task files that lack 'Files to Modify or Files to Create' and 'Implementation Notes' sections. Task 1 (create-branch) has: Repository, Target Branch, Bookend Type, Description, Acceptance Criteria, Test Requirements, Dependencies — missing Files to Modify/Create and Implementation Notes. Task 9 (merge-branch) similarly lacks these sections. The assertion does not exempt bookend tasks from the non-documentation template requirements."

Pass rate: 99% · Tokens: 75,128 · Duration: 859s

Baseline (169750fe): 99% · 120,638 tokens · 240s


Generated by sdlc-workflow/run-evals v0.13.10

@ruromero ruromero added ok-to-test Enable verify-pr execution on PRs from forks and removed ok-to-test Enable verify-pr execution on PRs from forks labels Oct 27, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

🤖 Finished Verify Pr · ❌ Failure (ensuring provider "vertex-ai": provider create "vertex-ai" failed: exit status 1 (output: Error: × code: 'Client specified an invalid argument', message: "provider │ credentials are not declared by pr…) · Started 6:41 PM UTC · Completed 6:41 PM UTC

Commit: 79bd354 · View workflow run →

Effort: high

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

🤖 Finished Verify Pr · ❌ Failure (ensuring provider "vertex-ai": provider create "vertex-ai" failed: exit status 1 (output: Error: × code: 'Client specified an invalid argument', message: "provider │ credentials are not declared by pr…) · Started 6:49 PM UTC · Completed 6:49 PM UTC

Commit: 79bd354 · View workflow run →

Effort: high

@ruromero
ruromero requested a review from mrizzi October 27, 2026 18:55
Trace shared artifacts through primary, fallback, on-demand, and consumer paths. Add eval coverage and record the constraint.

Implements TC-7146

Assisted-by: Claude Code
Require eval plans to test stable identifiers and existing update behavior.

Implements TC-7146

Assisted-by: Claude Code
Require path enumeration in impact maps and keep excluded writers out of task scope until clarified.

Implements TC-7146

Assisted-by: Claude Code
Require the multi-writer eval to check documentation of every population path.

Implements TC-7146

Assisted-by: Claude Code
@github-actions github-actions Bot removed the ok-to-test Enable verify-pr execution on PRs from forks label Oct 28, 2026
@ruromero

Copy link
Copy Markdown
Collaborator Author

[sdlc-workflow/verify-pr] Re: @sourcery-ai review — Classified as suggestion — the review provides a general risk caveat and requests human review, but does not request a specific code change. No sub-task created.

@ruromero

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-7146 (commit 5270a81)

Check Result Details
Review Feedback PASS One review-body suggestion was classified; no code-change requests or sub-tasks.
Root-Cause Investigation N/A No review, CI, or eval-regression sub-tasks were created.
Scope Containment WARN All 3 task-listed files are present; CLAUDE.md and docs/constraints.md are additional changes.
Diff Size WARN 119 lines across 5 files versus 3 task-listed files; modest line count, but broader file scope.
Commit Traceability PASS All 4 commits reference TC-7146.
Sensitive Patterns PASS No sensitive patterns found in 118 added lines.
CI Status PASS Plugin validation, Skillsaw, Python tests, and Eval PR Run passed; verify-pr jobs were skipped.
Acceptance Criteria PASS 3/3 met; eval-7 passed 10/10 and eval-8 passed 4/4.
Test Quality WARN Eval Quality: 103/104 (99%); eval-5’s single bookend-template failure matches the baseline and is pre-existing. No regressions. Repetitive-test and test-documentation checks passed.
Test Change Classification ADDITIVE Two eval cases and one synthetic fixture added; no coverage removed or weakened.
Verification Commands PASS uvx skillsaw had no error-level findings; plugin validation passed.

Overall: WARN

The requested shared-artifact guidance and positive/conflicting-scope evals pass. The warnings are the two additional files outside the Jira task’s listed file set and one pre-existing eval-5 assertion failure.


This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10.

@mrizzi mrizzi added the ok-to-test Enable verify-pr execution on PRs from forks label Oct 28, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

🤖 Finished Verify Pr · ✅ Success · Started 12:41 PM UTC · Completed 12:49 PM UTC

Commit: 5270a81 · View workflow run →

Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $4.38

@fullsend-ai-review

Copy link
Copy Markdown

Verification Report for TC-7146 (commit 5270a81)

Check Result Details
Review Feedback PASS One review-body suggestion (sourcery-ai) was classified in a prior run; no code-change requests and no sub-tasks.
Root-Cause Investigation N/A No review, CI, or eval-regression sub-tasks were created, so there was nothing to investigate.
Scope Containment WARN All 3 task-listed files present; docs/constraints.md (records constraint 1.106 — justified by the task) and CLAUDE.md (unrelated Jira Field Defaults + Serena example) are additional changes.
Diff Size PASS 118 insertions / 1 deletion across 5 files vs 3 task-listed files — proportionate for a docs + eval task.
Commit Traceability PASS All 4 commits reference TC-7146 (Implements TC-7146).
Sensitive Patterns PASS No secrets or credentials in the 118 added lines.
CI Status PASS All head-SHA check-runs succeeded (Plugin Validation, Skill Lint, Script Unit Tests 3.11-3.14, Sourcery, Trigger Eval Dispatch) and the Eval PR Run status is success.
Acceptance Criteria PASS 3/3 met — verified from the PR diff and the passing Eval PR Run; new evals 7 (10/10) and 8 (4/4) pass 100%.
Test Quality WARN Eval Quality 99%; eval-5's single bookend-task assertion failure matches the baseline and is pre-existing (no regressions). Repetitive-test and test-documentation checks: N/A (no unit-test functions changed).
Test Change Classification ADDITIVE Two eval cases and one synthetic fixture added; no existing coverage removed or weakened.
Verification Commands PASS uvx skillsaw -> Skill Lint (success); claude plugin validate -> Plugin Validation (success). Not runnable in the sandbox; CI equivalents used as evidence.

Overall: WARN

The requested shared-artifact planning guidance (Step 3 producer/consumer analysis and Step 5 task-decomposition coverage), the new constraint 1.106, and the two new eval cases (positive full-coverage eval-7 and conflicting-scope eval-8) are all present and pass. The warnings are non-blocking: two files were changed outside the task's listed set (docs/constraints.md is justified by the task's constraint-recording note; CLAUDE.md's Jira Field Defaults / Serena example appear unrelated to TC-7146 and are worth confirming), and one pre-existing eval-5 bookend-task assertion failure that also fails at baseline. No code-change requests, CI failures, or eval regressions were found, so no sub-tasks were created. This re-run verifies the same commit as a prior report; the per-commit comment is updated in place.


This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10.

@mrizzi
mrizzi merged commit 6328c3a into RHEcosystemAppEng:main Oct 28, 2026
23 checks passed
@ruromero
ruromero deleted the TC-7146 branch October 28, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Enable verify-pr execution on PRs from forks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants