Skip to content

feat: add lambda-runtime-invocation-id header - #1159

Merged
darklight3it merged 13 commits into
mainfrom
feat/add-runtime-invocation-id-header-support
Sep 2, 2026
Merged

feat: add lambda-runtime-invocation-id header#1159
darklight3it merged 13 commits into
mainfrom
feat/add-runtime-invocation-id-header-support

Conversation

@darklight3it

@darklight3it darklight3it commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Add Lambda-Runtime-Invocation-Id header support for cross-wiring protection.

The RIC now echoes the invocation ID received from RAPID on /next back on /response and /error, enabling RAPID to detect and reject stale responses from timed-out invocations.

Problem

On Lambda Managed Instances (LMI) and On-Demand (OD), when an invoke times out, the runtime process continues running in the background. If a new invoke arrives with the same requestId, RAPID accepts it. The still-running old invocation eventually posts its response, and RAPID matches it to the new invoke — delivering the wrong response (cross-wiring).

Solution

RAPID sends a unique per-invoke nonce via Lambda-Runtime-Invocation-Id header on /next. The runtime echoes it back on /response and /error. RAPID validates the match before accepting the response.

Backward Compatibility

Fully backward compatible in both directions:

  • If RAPID doesn't send the header → RIC doesn't see it → doesn't echo → no behavior change
  • If RIC doesn't echo it (old version) → RAPID skips validation → no behavior change

Rate-limited malformed-header logging

Added an internal generic RateLimiter for runtime different uses. In this case the request was to rate limit warning caused by malformed Lambda-Runtime-Invocation-Id headers.

The limiter:

  • Uses a monotonic Instant and configurable Duration.
  • Is protected by a standard mutex because the check-and-update operation must be atomic across concurrent Tokio tasks.
  • Is process-local, so a static limiter is shared across warm invocations in the same Lambda execution environment.
  • Resets its timestamp and clears mutex poisoning if the mutex is poisoned, since the limiter state is disposable and must not crash the runtime.
  • Keeps rate-limiting independent from tracing, allowing a future tracing-subscriber integration without coupling the runtime behavior to a specific subscriber.

Malformed invocation-ID headers are ignored rather than converted with from_utf8_lossy, because the value may be echoed back as an HTTP header and must remain a valid ASCII header value. A rate-limited warning is emitted with the configured interval included as structured log metadata.

Testing

  • added unit test
  • added a new dockerized test working with the new RIE.

@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from b60f865 to 048c1e8 Compare August 9, 2026 18:16
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from 4c28d43 to afb6708 Compare August 26, 2026 16:15
@darklight3it
darklight3it marked this pull request as ready for review August 26, 2026 16:18
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from afb6708 to de1f2e4 Compare August 26, 2026 16:23
@darklight3it darklight3it added the enhancement New feature or request label Aug 27, 2026
@darklight3it darklight3it self-assigned this Aug 27, 2026
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch 5 times, most recently from c403f80 to 31c22d2 Compare August 27, 2026 12:49
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from 31c22d2 to 2cabdac Compare August 27, 2026 13:11
Comment thread lambda-runtime/src/types.rs Outdated
Comment thread examples/invocation-id-concurrent/src/main.rs Outdated
Comment thread test/dockerized/scenarios/concurrent_scenarios.py
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from fc33a71 to 1152b7c Compare August 27, 2026 16:04

@jlizen jlizen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core approach is good, but some small tweaks.

Also: currently we log Lambda function timeout! for any 410, but we now will have a 410 if a stale response is rejected. We should tweak the message.

Comment thread lambda-runtime/src/requests.rs
Comment thread examples/invocation-id-concurrent/src/main.rs
Comment thread test/dockerized/scenarios/concurrent_scenarios.py Outdated
Comment thread test/dockerized/scenarios/concurrent_scenarios.py Outdated
Comment thread lambda-runtime/src/requests.rs
Comment thread lambda-runtime/src/layers/api_response.rs
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch 2 times, most recently from 58592a7 to bc1209f Compare August 31, 2026 08:01
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from bc1209f to c09906e Compare August 31, 2026 08:06
@darklight3it
darklight3it force-pushed the feat/add-runtime-invocation-id-header-support branch from c09906e to eddf4c5 Compare August 31, 2026 10:52
Comment thread examples/invocation-id-concurrent/Cargo.toml Outdated
Comment thread lambda-runtime/src/layers/api_response.rs Outdated
@darklight3it
darklight3it merged commit 31be935 into main Sep 2, 2026
31 checks passed
@darklight3it
darklight3it deleted the feat/add-runtime-invocation-id-header-support branch September 2, 2026 08:59
@darklight3it darklight3it mentioned this pull request Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants