Skip to content

feat(guardrails): report provider errors on guardrail validation results [AL-673] - #1956

Open
apetraru-uipath wants to merge 1 commit into
mainfrom
fix/al-673-guardrail-error-type
Open

apetraru-uipath wants to merge 1 commit into
mainfrom
fix/al-673-guardrail-error-type

Conversation

@apetraru-uipath

@apetraru-uipath apetraru-uipath commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

The guardrail validate API answers PROVIDER_ERROR, INTERNAL_ERROR or RATE_LIMIT_EXCEEDED when it could not evaluate the input. evaluate_guardrail turned these into VALIDATION_FAILED through its unknown-result fallback, so callers could not tell a real violation from a validation that never ran.

Prod impact (SRE-687087): on 2026-10-22, an LLM-as-Judge model rejected temperature with a 400. 90 eval runs were reported as guardrail violations.

Changes:

  • GuardrailValidationResult.error_type (alias errorType) uses a new GuardrailValidationErrorType enum: provider_error, internal_error or rate_limit_exceeded.
    • It is set only for those backend results, matched case-insensitively.
    • It is None for every other result.
  • result stays validation_failed, so the guardrail still fails closed. No consumer changes are needed:
    • Every existing == / != VALIDATION_FAILED check, here and in uipath-langchain (middlewares, adapter, escalate action, guardrail nodes), still fires its action.
    • Adding new values to GuardrailValidationResultType was rejected on purpose. Those same checks would have started letting runs through, and released uipath-langchain versions would pick up a newer platform.
  • Logging: an error result and an unknown result each log a warning.
  • Imports: the enum is exported from uipath.core.guardrails only. It is not added to the uipath.platform.guardrails re-export block, because that block is marked for removal in 2.3.0.

Versions:

  • uipath-core 0.5.34.
  • uipath-platform 0.2.42, which requires core ≥ 0.5.34.
  • The uv.lock files are refreshed.

Follow-ups:

  • uipath-langchain will label these failures in the guardrail validation details ("Guardrail could not be evaluated (provider_error): …") once this is released.
  • The backend fix for the root cause is UiPath/Agents#6797.

Ticket: AL-673

How has this been tested?

  • uipath-core guardrail tests: 51 passed. The full suite is 246 passed. They cover:
    • error_type defaults to None.
    • The errorType alias round-trips.
    • The enum values.
  • uipath-platform service tests: 59 passed. The full suite is 1898 passed and 7 skipped (the live LLM tests). They cover:
    • Each error result gives validation_failed, the matching error_type, and the backend details kept as reason. This holds for lowercase results too.
    • An unknown result gives validation_failed, error_type=None and a warning.
    • PASSED and VALIDATION_FAILED give error_type=None.
  • ruff check, ruff format --check and mypy are clean on both packages.
  • check_version_uniqueness.py and check_dependency_version_bumps.py pass, and uv lock --locked is current.

🤖 Generated with Claude Code

…lts [AL-673]

The guardrail validate API answers PROVIDER_ERROR, INTERNAL_ERROR or
RATE_LIMIT_EXCEEDED when it could not evaluate the input, for example when
an LLM-as-Judge model rejects the request. evaluate_guardrail turned these
into VALIDATION_FAILED through its unknown-result fallback, so callers could
not tell a real violation from a failed evaluation (SRE-687087: judge models
that deprecate `temperature` answered 400 and 90 eval runs were blocked).

GuardrailValidationResult gains an additive error_type
(GuardrailValidationErrorType) that is set for those results. result stays
validation_failed, so every existing VALIDATION_FAILED check keeps firing
its action (fail closed); error_type only says why the validation did not
complete. Error and unknown results are now logged as warnings.

Bumps uipath-core to 0.5.34 and uipath-platform to 0.2.42 (requiring
uipath-core>=0.5.34) and refreshes the uv.lock files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 25, 2026 03:59
@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-runtime test:uipath-integrations labels Oct 25, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The additive metadata preserves existing failure handling, has focused regression coverage, and leaves no unresolved blocking findings.

No open issues 🎉

What changed in this PR

This PR adds error metadata to guardrail validation results so callers can distinguish service failures from actual violations while preserving existing fail-closed behavior.

Changes:

  • Adds and exports GuardrailValidationErrorType with an optional error_type result field.
  • Maps backend errors case-insensitively and logs warnings for errors and unknown results.
  • Adds regression tests and updates package versions, dependency requirements, and lockfiles.
File Description
packages/​uipath/​uv.lock Updates locked core and platform versions.
packages/​uipath-platform/​uv.lock Updates locked core and platform versions.
packages/​uipath-platform/​tests/​services/​test_guardrails_service.py Tests error mapping, warnings, and preserved failure behavior.
packages/​uipath-platform/​src/​uipath/​platform/​guardrails/​_guardrails_service.py Classifies backend errors and attaches metadata while failing closed.
packages/​uipath-platform/​pyproject.toml Bumps platform version and minimum core dependency.
packages/​uipath-core/​uv.lock Updates the locked core version.
packages/​uipath-core/​tests/​guardrails/​test_guardrails_models.py Tests enum values, defaults, aliases, and serialization.
packages/​uipath-core/​src/​uipath/​core/​guardrails/​guardrails.py Defines the error enum and optional result field.
packages/​uipath-core/​src/​uipath/​core/​guardrails/​__init__.py Exports the new error enum.
packages/​uipath-core/​pyproject.toml Bumps core to version 0.5.34.

Review effort: Balanced

@sonarqubecloud

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-integrations test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants