feat(server): Report sanitized diagnostics on upstream 4xx rejections [SAO-17523]

[josjeon]

Summary

When the upstream authorization service rejects a check, the log line records only the operation and the status code. This PR attaches sanitized, value-free diagnostics to it so the cause is identifiable from logs alone.

What happened

During the 2026-09-23 multitenant incident, POST /api/v1/auth/runtime-token-exchange returned 502. Direct pod inspection showed the sequence, twice:

16:39:41.943  POST /internal/auth/agent_control/check_management_access -> 422
16:39:41.948  Authorization upstream rejected operation control_bindings.read with status 422
16:39:41.956  Request completed: POST /api/v1/agents/initAgent -> 502

16:39:42.130  POST /internal/auth/agent_control/check_management_access -> 422
16:39:42.135  Authorization upstream rejected operation runtime.token_exchange with status 422
16:39:42.142  Request completed: POST /api/v1/auth/runtime-token-exchange -> 502

That second line is the log this PR changes. It carries the operation and the status, and nothing else, so the rejected field was never identified. The root cause of that incident remains unconfirmed for exactly this reason.

What changed

The 400 <= status < 500 branch of HttpUpstreamAuthProvider._handle_response now logs three additional fields:

Field	Content
operation / status_code	As before, now structured rather than only interpolated
target_context	Presence, and the shape of each target field
upstream_validation	type and loc per entry, capped
upstream_validation_total	Total entries, so a capped list is not mistaken for complete

Orbit answers with the standard FastAPI validation envelope, and type plus loc alone separate the plausible causes:

Cause	type	loc
Malformed target_id	uuid_parsing	body.context.target_id
Unsupported target_type	enum	body.context.target_type
Unknown operation	enum	body.operation

What is deliberately not logged

• input, ctx, and msg from each validation entry. All three echo caller-supplied values.
• Target identifiers. _field_shape reports null, empty, non_string:<type>, or string:len=N instead of the value.

The length is kept because whether a target_id was UUID-shaped is precisely the question the incident could not answer. That is a deliberate tradeoff against the no-sensitive-data goal, and the conservative alternative would be a bare string.

Scope

• Changed files:
  • server/src/agent_control_server/auth_framework/providers/http_upstream.py
  • server/tests/test_auth_framework.py
  • server/src/agent_control_server/services/control_bindings.py, one type annotation. Unrelated to this feature: SQLAlchemy 2.1.0 stopped letting mypy infer the element type through result.scalars().all(), so CI began failing typecheck on an unchanged line. The locked 2.0.51 still infers it, which is why it only shows in CI. Reproduced locally with mypy==2.3.1 and sqlalchemy==2.1.0, and the annotation holds under both versions.
• No behavior change. The APIError, its status, and its message are untouched; this is additive logging.
• Applies to every operation's 4xx path, not only runtime-token exchange.

Out of scope:

• Local request validation on the runtime-token endpoint. That is SAO-17522, in fix(server): Reject malformed runtime-token target context before authorization [SAO-17522] #268.
• Any Orbit-specific validation. Agent Control stays OSS-generic and target values remain opaque.
• Identifying the caller that produced the original 422. These diagnostics are a prerequisite for that, not a substitute; the parent story's criterion stays open until the issue recurs with this logging in place or an equivalent reproduction is found.

Risk and Rollout

• Risk level: low. Additive logging on an error path that already existed.
• The diagnostic parser never raises. A non-JSON body, a non-object body, a detail that is not a list, and a non-object entry all degrade to an empty summary rather than turning a 502 into a 500. Each case has a test.
• No migration or configuration change.
• Rollback plan: revert this PR.

Testing

• Added automated tests: 6 new cases covering the realistic incident shape, all four target-context shapes, sentinel-based leak checking, four malformed upstream bodies, and the truncation cap.
• Proved load-bearing: reverting the log line to its previous form fails 10 of the new cases.
• server/tests/test_auth_framework.py: 134 passed.
• Full server suite: 912 passed.
• ruff check server/src and mypy server/src clean (51 files).
• Validated in multitenant staging: not possible from a PR branch. docker-publish.yml only pushes images on main or a v* tag, so there is no deployable image until this merges.

Note: server/Makefile pins AGENT_CONTROL_DB_PORT=5432 inside TEST_DB_ENV, so the port must be overridden when another Postgres already holds 5432.

Checklist

• Linked issue: SAO-17523, a subtask of SAO-17418.
• Documentation/examples: no update required; no API or configuration surface changes.
• Independent of fix(server): Reject malformed runtime-token target context before authorization [SAO-17522] #268. Different file, no overlap, either order merges cleanly.

AI Tool Assistance Usage Statement

• AI assistance was used to draft parts of the implementation, that was subsequently modified and extended.
• AI assistance was used in generating tests/documentation/comments for this change.
• AI assistance was used for optimizing/troubleshooting/refactoring existing code in this change.
• AI assistance was used to draft this entire change as is.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!