feat(middleware): add a hook to inspect HTTP responses

[pimlock]

Summary

Let configured middleware inspect, transform, or block upstream HTTP responses before they reach the sandbox. This implements the response hook defined in #3073, including whole-body inspection that can withhold a response and streaming inspection that can stop further delivery.

State-machine walkthrough: diagrams and transition tables explain middleware channel lifetime, fail-open/fail-closed behavior, skip_remaining, and the HTTP commitment boundary.

Related Issue

Part of #2691. Builds on merged #3072 and #3073.

Changes

• Run response stages in policy order with independently selected header-only, whole-body, or streaming inspection. The runtime owns sequencing, limits, failure handling, and stage cleanup; the HTTP relay owns buffering, framing, and delivery.
• Keep stage termination separate from response termination. skip_remaining and fail-open disable only the current stage while later stages continue. A valid block stops the response regardless of failure policy.
• Return a canonical 403 for a block or 502 for a middleware failure handled fail-closed before final headers are sent. After commitment, stop delivery without appending error bytes or completing the response framing. Whole-body inspection delays commitment and uses a shared, non-resetting 120-second accumulation deadline local to the relay.
• Extend the content-guard example to inspect complete response bodies with the same configured-literal matching semantics used for requests and WebSocket messages. Matches can be redacted or blocked; unavailable whole-body inspection follows the configured failure policy.
• Add runtime and relay regression coverage, external-middleware smoke coverage, and documentation updates. Cross-unit matching, spooled whole-body storage, and the separate protocol demonstration remain follow-up work.

Testing

Live middleware test results and performance measurements: single-stage and mixed-mode chains, body modes and actions, measured overhead, and Content-Length fix validation.

Validation reported for the implementation:

• mise run pre-commit passed.
• Content-guard unit tests passed: 15 tests via cargo test --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --offline.
• Docker content-guard smoke suite passed response pass-through, redaction, denial, and guarded/unguarded requests through external middleware. Shell and Python fixture syntax checks passed.
• Full mise run test and mise run ci.

Checklist

• Follows Conventional Commits.
• Commits are signed off for DCO.
• Unit and Docker smoke coverage added/updated.
• Architecture and example documentation updated.
• Full branch validation is green.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3074.docs.buildwithfern.com/openshell

[github-actions]

Label test:e2e applied for 150ae32. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[pimlock]

gator-agent

PR Review Status

This implementation is project-valid under accepted issue #2691, and the initial full-diff review found one blocking fail-open correctness issue. The response middleware, relay coverage, and Fern documentation otherwise align with the intended response-hook path.

Action required: update the response fallback so valid but middleware-unrepresentable responses obey each selected stage's on_error policy, then push a new head for a delta review.

Blocking findings:

• GATOR-150ae32b-01: valid responses can become unconditional 502 failures even when every selected response stage is fail_open.

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Implements the accepted HTTP response middleware work in #2691 and builds on merged #3072 and #3073.
• Docs: Relevant Fern documentation and operator-facing skill guidance are updated.
• Checks: DCO and currently dispatched standard checks are green; required E2E has not run for this head.
• E2E: test:e2e is applied, but the label-help workflow requires a maintainer to re-run all jobs after review feedback is resolved.
• Head SHA: 150ae32b4293654b34ab83f71fd01c3e647490aa
• Base SHA: 48c449d8c80b8ad0201d24bef5c2ece3be233580
• Merge base SHA: 48c449d8c80b8ad0201d24bef5c2ece3be233580
• Patch ID: 274606ce582cd5701da75ddda5ae6c71452240d2
• Gator payload: 8
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[pimlock]

gator-agent

PR Review Status

The follow-up delta fixes the fail-open response fallback: valid responses that cannot fit the middleware protocol now follow each selected stage's on_error policy, while malformed or unsafe HTTP remains fail closed. No blocking findings remain, and the addressed review thread is resolved.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-150ae32b-01: resolved by the current head; obs-text and envelope-limit coverage now verifies both fail-open passthrough and fail-closed 502 behavior.

Gator metadata

• Validation: Implements accepted issue #2691 and builds on merged #3072 and #3073.
• Docs: Fern middleware documentation and sandbox architecture guidance cover the corrected behavior.
• Checks: Current-head required checks are queued or running; completed DCO and Trivy gates are green.
• E2E: test:e2e is applied and the current-head E2E workflow/check is dispatched.
• Head SHA: 408382aa75381855ff8df0fdff684a7de7a3a0eb
• Base SHA: 48c449d8c80b8ad0201d24bef5c2ece3be233580
• Merge base SHA: 48c449d8c80b8ad0201d24bef5c2ece3be233580
• Patch ID: b646c493d45e19d8519c2edc648f93d782498570
• Gator payload: 8
• Review mode: follow_up
• Previous reviewed SHA: 150ae32b4293654b34ab83f71fd01c3e647490aa
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:watch-pipeline