Repository navigation
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replaces the single escalation bullet with a dedicated section covering five cases: opening request to contact support, in-scope-but-unanswerable questions, broad/ambiguous questions (clarify using only available topics), out-of-scope questions (never offer a resource), and users stuck after repeated attempts. Fixes the bug where an unrelated retrieved document's contact info leaked into unrelated answers (e.g. "What is the definition of ABBR?"). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Creates a synthetic HR KB sandbox (hr-kb-sandbox/) to manually exercise the 5 escalation scenarios and the ABBR regression fix in the HR agent instructions: opening support requests, in-scope-unanswerable questions, broad/multi-topic questions, out-of-scope questions (using the existing IT KB sandbox), and users stuck after reasonable attempts. All 10 eval cases pass against the instructions text as written. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers branches the first pass didn't exercise: a stuck user with no retrievable resource (rule 5 doesn't override the grounding rule), a broad question with zero retrieval rather than several sub-categories (falls to rule 2, not rule 3), an HR/IT boundary question that stays in scope because an approved HR source answers it, and an opening message that already states its subject. 13/14 pass; the last one surfaces a genuine gap in rule 1, which has no carve-out for a self-describing opening message — documented as a finding with a suggested (not yet applied) wording fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Case 14 in the eval results showed rule 1 re-asking "describe the issue" even when the opening message already named the subject (e.g., "open a case about my LOA request not showing up"). Adds a carve-out so the agent proceeds straight to rules 2-5 for that subject instead. Trimmed other bullets to stay under the 8000-char budget (7992 final). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Aligns the ServiceNow HR case-creation topic's trigger description with the escalation rules in the main HR agent instructions: adds a resolvability gate (don't trigger when the subject is already answerable from HR knowledge sources, unless the user insists after an answer) and a context-gathering gate (don't trigger on a bare "create an HR case" with no subject — wait for the opening-request clarification turn). Deliverable is plain text for pasting into the live Copilot Studio topic configuration; the topic itself lives outside this repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot Studio topic descriptions cap at 1024 characters; the prior draft (2141 chars) exceeded it. Condensed wording while keeping all five gating conditions, the explicit-insist override, and one example each for valid/invalid. Final: 1016 chars. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live DA testing showed the prior wording didn't prevent a bare "create an HR case" from triggering the topic immediately, which then captured the next message and went straight to case creation with no resolvability check possible (DA has no supported in-topic knowledge- retrieval node, confirmed against RULE-007). Re-ordered the description to state the subject-presence requirement as the first, unambiguous condition instead of one bullet among several, so the orchestrator has a better chance of honoring it on the initial trigger decision — the only point where it can be enforced. Documented as a probabilistic mitigation, not a guarantee, in the design doc. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…erable) Scoped down from the earlier full-rewrite discussion to just the escalation changes, applied to the OOB baseline instructions (tools/ess-ca-to-da/reference/it/agent.yml) rather than the richer DuPont-customized draft. Written to a standalone text file instead of editing the yml, per explicit instruction. Changes (verified via diff against the OOB original, nothing else touched): added a consolidated "Support resources and escalation" section (5 rules, mirroring the HR agent's), folded the old standalone "Abstain with" line and "EAP/People team" mention into it, and aligned the Dupont-problem/broad-question clarification to the HR approach (grounded in retrieved sub-categories, not a pre-emptive generic list). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Redirected from the earlier OOB-baseline pass to this richer, DuPont-customized prompt per explicit request. Added a new "Support resources and escalation" section after the existing "Escalation behavior" section: covers opening support requests, in-scope- unanswerable questions, out-of-scope questions, and users stuck after reasonable attempts as new rules; the broad/ambiguous-question scenario is cross-referenced rather than duplicated, since this file already has the most mature version of that logic in the repo (Core principle, Conversational interaction style, Clarifying questions sections). The existing "Escalation behavior" section (when to use the standing ServiceNow portal) is untouched and complementary — the new section's grounding rule governs only named/specific contacts, not that standing destination. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The "clarify first for broad/multi-category requests" rule was stated, in different wording, in 7 places. Trimmed each location down to just its unique contribution (tone/examples, retrieval-specific operationalization, when-to-clarify criteria, response-formatting discipline, procedural answer templates), keeping "Core principle" as the sole canonical statement and the retrieval-clustering bullet in "Authoritative knowledge source" as its only operational detail. Removed the "Configurable behavior preference" section entirely since it added nothing beyond what's now stated once. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds the IT equivalent of the HR eval suite, using the existing
dupont-kb-sandbox (which had no paired eval-cases/results files yet).
14 cases covering the 5 required scenarios, a regression check
analogous to the HR agent's ABBR bug, and 4 negative/edge cases.
14/14 pass on the first manual walkthrough — unlike HR, no instruction
gap was found, since the rule 1 carve-out ("unless already stated") and
the unrelated-document-leak grounding rule were written into this
prompt proactively from the start, informed by what testing caught in
the HR agent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds 12 cases (15-26) derived from a real UAT test pass by Liliana Martinez against the pre-update "Gold Prompt" — not this repo's revised dupont-ess-it-agent-prompt.md. Each maps the tester's observed failures (not asking a clarifying question before a multi-category answer, and one hallucinated MFA option, "OKTA," not present in any KB) onto our existing KB sandbox and checks whether the current prompt would have avoided them. 8/12 are clean passes under literal application; 2 (nested password drill-down, VPN issue-type clarification) are marked "predicted pass" rather than flat pass since they're the exact failure modes observed live and warrant real re-verification, not just this manual walkthrough. 2 are sandbox-fidelity gaps (peripherals breadth, printer OS-version variant) needing a richer KB to test. 1 (citation rendering/KB-number bug) is a platform/connector issue no prompt wording can fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Second dedup pass: merged "Conversational interaction style" and
"Clarifying questions" into one section, absorbing the single/multi-
cluster retrieval bullet from "Authoritative knowledge source" and the
generic-request procedural template from "Answer format". Fixes a
duplicate printer example (same example lived in two sections) and two
remaining restatements (an inverse of Core principle's closing
sentence, and a "present all options" bullet redundant with the merged
section).
No requirement dropped — every criterion, example, and formatting rule
survives, just stated once. Added explicit sequential-narrowing
language ("ask again if the narrowed request still spans distinct
sub-categories") addressing the real UAT gap where password reset
asked system but never followed up with known-vs-locked once narrowed
to Windows.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Checked commit bc500ac's diff against the design doc's "no requirement dropped" claim and re-ran all 26 eval cases against the consolidated text. Two minor findings noted (two criteria folded into the broader trigger rather than kept as literal bullets; one worked example's "reflect + something else" modeling simplified) — neither changes any case's expected behavior. Case 15b (nested password drill-down) upgraded from "predicted pass" to "pass": the consolidated text now states that exact scenario as its own worked example, not just an inference. Case 16 (VPN) stays "predicted pass" since no VPN-specific example exists in the prompt. All other 24 cases unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removed the revision-history blockquote (meta-commentary, not an instruction) and tightened three sentences (identity paragraph, escalation-info list intro). No requirement, rule, or example dropped. Final: 7987 chars (was 8264). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Diffed the entire rule-bearing body of the prompt (every section the 26 eval cases actually test) between the pre-trim and post-trim versions. Only one wording-only line differs (the escalation-info list intro), untested by any case. All 26 cases retain their prior verdicts unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot Studio's Instructions box counter ran ahead of a plain LF character count (7999/8000 reached before the end of a 7994-char file) -- most likely because the box counts CRLF line breaks (2 chars) where a plain count uses LF (1 char); 70 lines accounts for nearly the full gap. Trimmed several sentences for brevity without dropping any rule, keeping the new MUST-search-before-case-creation wording in rule 1 intact. Final: 7922 chars (plain count), with margin for the CRLF overhead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Packages the manual curation process developed this session (5-scenario
escalation pattern, dedup-and-verify discipline, 8000-char budget
trimming with CRLF-counting awareness, paired eval-sandbox maintenance,
UAT-feedback ingestion, runtime-feasibility flagging) as two skills in
this product's existing {category}/{verb}/SKILL.md convention:
solutions/ess-maker-skills/src/skills/instructions/curate-hr and
curate-it.
Scoped to curation only, not live deployment: push.py's classify_path
has no path for an agent's instructions field today, and extending a
shared deployment script is called out as a follow-on, not built here.
Two separate skills per explicit choice, with the shared process
written independently into each file and agent-specific facts (target
file, standing-vs-named resource model, sandbox location) baked in as
content.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Addresses findings from the local-branch code review:
- [BLOCKER] docs/plans/assets/dupont-kb-sandbox/ (23 synthetic KB
articles) was never committed despite being a hard dependency of
dupont-it-eval-cases.md, dupont-it-eval-results.md, and both new
curate-hr/curate-it skill files. Added now so the eval suite is
actually reproducible from a fresh checkout.
- [MEDIUM] Replaced a real individual's name ("Liliana Martinez") with
a role reference ("UAT reviewer") in dupont-it-eval-cases.md, per
this repo's own role-not-name convention used elsewhere.
- [LOW] Committed the previously-untracked convenience .txt/.md copies
(dupont-ess-it-agent-prompt.txt, hr-agent-instructions-support-
escalation.txt, hr-agent-test-prompts.md) instead of leaving them as
uncommitted duplicates — verified each is byte-identical to its
already-committed counterpart before committing, so no stale content
was introduced.
The unrelated 2026-09-29 ESS instruction hill-climb files remain
untracked, out of scope per explicit instruction.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both skills previously assumed every incoming requirement maps onto one of the 5 escalation scenarios. Added a classification step before applying the pattern: requirements that extend/modify a known scenario follow the validated worked pattern (now Step 3a); a genuinely new rule category (tone, formatting, citation rules, new eligibility/routing rules, etc.) instead gets general editorial guidance (find the most structurally similar section, hold it to the same grounding discipline, tell the maker it's unvalidated) while still running the general-purpose steps (dedup, char budget, eval sandbox, runtime-feasibility flag, approval) unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hr-kb-sandbox/,dupont-kb-sandbox/) with eval-cases/eval-results files covering all 5 scenarios, regression checks, and negative/edge cases — including cases derived from real UAT tester feedback against the IT agent's "Gold Prompt".solutions/ess-maker-skills/src/skills/instructions/{curate-hr,curate-it}/SKILL.md, packaging this curation process (escalation pattern application, dedup discipline, character-budget enforcement, eval-sandbox maintenance, runtime-feasibility flagging) for future requirement changes — explicitly scoped to curation only, not live deployment.Test plan
docs/plans/assets/{hr-agent,dupont-it}-eval-results.md(PASS / PREDICTED PASS / SANDBOX GAP / OUT OF SCOPE verdicts) — review these before mergedocs/prompts/2026-10-15-hr-agent-instructions-support-escalation.md) and IT instructions (docs/plans/assets/dupont-ess-it-agent-prompt.md) into their respective live Copilot Studio Instructions boxes and spot-check the character counter🤖 Generated with Claude Code