BE-868: Set up modular REST APIs and shared OpenAPI documentation - #9770
TimDiekmann wants to merge 10 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 4 Skipped Deployments
|
PR SummaryMedium Risk Overview Adds versioned Entities v1, Types v1, and Internal APIs (caller endpoints for now) with Aide/Schemars OpenAPI generation, RFC 9457 Problem Details in specs, and insta snapshots per API. Public APIs accept service delegation and Cloudflare Access; Internal and legacy also accept Kratos sessions, wired through separate provider chains and audience-specific credential documentation. Replaces the optional CDN utoipa-scalar viewer and Supporting changes: Reviewed by Cursor Bugbot for commit 643cf52. Bugbot is set up for automated code reviews on this repo. Configure here. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #9770 +/- ##
==========================================
+ Coverage 66.58% 66.82% +0.23%
==========================================
Files 1805 1871 +66
Lines 191004 196421 +5417
Branches 7858 7988 +130
==========================================
+ Hits 127184 131260 +4076
- Misses 62323 63621 +1298
- Partials 1497 1540 +43
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will not alter performance
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
as_constant |
< 1 ns | < 1 ns | N/A | |
constant_equal |
< 1 ns | < 1 ns | N/A | |
constant_not_equal |
< 1 ns | < 1 ns | N/A | |
access |
< 1 ns | < 1 ns | N/A | |
runtime_equal |
< 1 ns | < 1 ns | N/A | |
runtime_not_equal |
< 1 ns | < 1 ns | N/A |
Comparing t/be-868-modular-rest-apis (643cf52) with main (3dba8b8)1
Footnotes
| probe, rate_limit, telemetry, | ||
| }; | ||
|
|
||
| pub struct Dependencies<S> |
There was a problem hiding this comment.
non-blocking: longer term we should really move away from Dependencies via Extension to just using state with https://docs.rs/axum/latest/axum/extract/trait.FromRef.html (State and FromRef were introduced after we had our first version) but it makes it possible to have things be actually type checked that their data is there.
There was a problem hiding this comment.
I remember when they added it, but I never got to it. I might just do it as follow-up for the new endpoints.
| prefix: &'static str, | ||
| audience: Audience, | ||
| info: Info, | ||
| create_routes: impl FnOnce() -> ApiRouter, |
There was a problem hiding this comment.
why FnOnce here? why not give it the ApiRouter directly?
There was a problem hiding this comment.
Because the routes have to be generated after build registers aide's on_error and inside the context that this finish_api drains.
If we'd pass an ApiRouter directly, this would not see the thread-local on_error. I was thinking about using fn() -> ApiRouter to prevent || prebuilt, but I found that to be overengineered.
| pub(super) fn problem_response(status: u16, description: &str) -> Response { | ||
| // Aide's shared generator uses the deserialization contract. Responses need the | ||
| // serialization contract; inline subschemas keep this generator's references local. | ||
| let mut generator = SchemaSettings::draft07() | ||
| .for_serialize() | ||
| .with(|settings| settings.inline_subschemas = true) | ||
| .into_generator(); | ||
| let mut schema = generator.subschema_for::<ProblemDetails<'static>>(); | ||
| let status_schema = schema | ||
| .pointer_mut("/properties/status") | ||
| .and_then(serde_json::Value::as_object_mut) | ||
| .expect("the problem schema should contain an object schema for status"); | ||
| status_schema | ||
| .retain(|key, _| !matches!(key.as_str(), "examples" | "format" | "minimum" | "maximum")); | ||
| status_schema.insert("const".to_owned(), status.into()); | ||
| status_schema.insert("examples".to_owned(), [status].into()); | ||
|
|
||
| if let Ok(status_code) = http::StatusCode::from_u16(status) | ||
| && let Some(reason) = status_code.canonical_reason() | ||
| { | ||
| schema | ||
| .pointer_mut("/properties/title") | ||
| .and_then(serde_json::Value::as_object_mut) | ||
| .expect("the problem schema should contain an object schema for title") | ||
| .insert("examples".to_owned(), [reason].into()); | ||
|
|
||
| let slug = reason.replace('\'', "").to_case(Case::Kebab); | ||
| let type_uri = format!("https://example.com/problems/{slug}"); | ||
| schema | ||
| .pointer_mut("/properties/type") | ||
| .and_then(serde_json::Value::as_object_mut) | ||
| .expect("the problem schema should contain an object schema for type") | ||
| .insert("examples".to_owned(), [type_uri].into()); | ||
| } | ||
|
|
||
| Response { | ||
| description: description.to_owned(), | ||
| content: IndexMap::from_iter([( | ||
| "application/problem+json".to_owned(), | ||
| MediaType { | ||
| schema: Some(SchemaObject { | ||
| json_schema: schema, | ||
| example: None, | ||
| external_docs: None, | ||
| }), | ||
| ..MediaType::default() | ||
| }, | ||
| )]), | ||
| ..Response::default() | ||
| } | ||
| } | ||
|
|
||
| pub(super) fn add_response(api: &mut OpenApi, status: u16, name: &str, response: Response) { | ||
| api.components | ||
| .get_or_insert_with(Default::default) | ||
| .responses | ||
| .insert(name.to_owned(), ReferenceOr::Item(response)); | ||
|
|
||
| let Some(paths) = &mut api.paths else { | ||
| return; | ||
| }; | ||
| for path in paths.paths.values_mut() { | ||
| let Some(path) = path.as_item_mut() else { | ||
| continue; | ||
| }; | ||
| for (_, operation) in iter_operations_mut(path) { | ||
| operation | ||
| .responses | ||
| .get_or_insert_with(Default::default) | ||
| .responses | ||
| .entry(StatusCode::Code(status)) | ||
| .or_insert_with(|| ReferenceOr::ref_(&format!("#/components/responses/{name}"))); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
I am a bit puzzled by this why the need for this? isn't this the exact case for OperationOutput and TransformOutput? This is what I use:
hash/libs/@local/graph/atlas/src/api/problem.rs
Lines 390 to 429 in adfb316
hash/libs/@local/graph/atlas/src/api/current.rs
Lines 61 to 63 in adfb316
| pub(super) fn responses(api: &mut OpenApi) { | ||
| for (status, name, description) in [ | ||
| ( | ||
| 400, | ||
| "MalformedCredentials", | ||
| "Malformed credentials or actor header.", | ||
| ), | ||
| ( | ||
| 401, | ||
| "AuthenticationRejected", | ||
| "The credentials cannot resolve to a permitted caller.", | ||
| ), | ||
| ( | ||
| 503, | ||
| "AuthenticationUnavailable", | ||
| "The credential provider or actor store is unavailable.", | ||
| ), | ||
| ] { | ||
| add_response(api, status, name, problem_response(status, description)); | ||
| } | ||
| } | ||
|
|
||
| pub(super) struct Actor<C>(ActorId, PhantomData<fn() -> C>); |
There was a problem hiding this comment.
I used .with and TransformOperation: https://github.com/hashintel/hash/blob/adfb31627ca77a9f8d0bc741c7e9b60b9921f666/libs/%40local/graph/atlas/src/api/clause.rs (https://docs.rs/aide/latest/aide/transform/struct.TransformOpenApi.html#method.with)
(note that the aide use rn is not ideal for errors in this case, something i am looking to remedy once we have aide inside all our other layers, needed to kinda paper over things as we didn't have aide support yet)
| } | ||
| } | ||
|
|
||
| pub(super) struct MaybeActor<C>(Option<ActorId>, PhantomData<fn() -> C>); |
There was a problem hiding this comment.
I guess we need this because Option only makes the parameters optional, not the security requirements: https://docs.rs/aide/latest/src/aide/impls/mod.rs.html#62-94
There was a problem hiding this comment.
Yes, and with that we also implicitly can add the empty security requirement.
| } | ||
| } | ||
|
|
||
| pub(super) fn responses(api: &mut OpenApi) { |
There was a problem hiding this comment.
this is my version:
hash/libs/@local/graph/atlas/src/api/clause.rs
Lines 63 to 74 in adfb316
There was a problem hiding this comment.
I had this before, I don't know why I lost this, and it annoys me. Fixed.
Give `problematic` and `hash-middleware` an `aide` feature so the authentication and rate-limit rejections document themselves as problem responses, and let each Graph API pull them in through `TransformOpenApi` clauses instead of hand-built response objects. Shared problem responses move into `components/responses`, each with an example taken from the runtime body. Tie each API's audience, security schemes and route markers together through a `Credentials` trait, and check after document generation that every required scheme is declared. Legacy routes and every API draw on one root `RateLimiters`; `with_principal_limits` stays as the override for later. Build the operator provider chain once and share it with the internal chain, answer unmatched paths with a problem document, panic on aide documentation defects while the routes register, and force the Scalar bundle patch at startup.
The limiter behind authentication budgets anonymous requests by address and actors across addresses, so `CallerLimitLayer`, `CallerRateLimitConfig` and `with_caller_limits` take the name of the `Caller` the provider chain resolves. A principal is an actor or an actor group and names neither outcome. The `stage` attribute of `hash.rate_limit.unchecked` follows with the value `caller`.
`SharedProviders` verify the credentials every API accepts, Cloudflare Access and service delegation, matching `shared_schemes` in the credential documentation. `SessionProviders` add the Kratos session in front of them for the APIs that serve browser sessions.
The provider chain used to consult the Cloudflare Access JWT ahead of service delegation, so a request carrying both resolved to the identity the edge stamped on it and never read the credential the caller presented on purpose. The providers now form three groups, `ExplicitProviders`, `SessionProviders` and `EnvironmentProviders`, which each router composes in that order. Only the Cloudflare Access provider is shared between the chains, for its JWKS cache.
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
🌟 What is the purpose of this PR?
Establish separate Graph REST APIs for Entities v1, Types v1, and Internal. Each API owns its routes and OpenAPI document, with shared documentation available through self-hosted Scalar at
/.🔗 Related links
🚫 Blocked by
None.
🔍 What does this change?
rest::routerthe Graph HTTP entrypoint, with legacy handlers underrest::legacyand shared authentication, middleware, telemetry, and probes owned byrest./entities/v1,/types/v1, and/internalmodules with optional and authenticated caller endpoints.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
The APIs currently contain caller endpoints for exercising authentication and documentation. Public authentication uses delegation and Cloudflare while OAuth is developed. Scalar's bundled JavaScript receives a checked patch for its optional-authentication indicator.
🐾 Next steps
Add the first entity and type operations, OAuth authentication, and an internal API versioning policy.
🛡 What tests cover this?
/.❓ How to test this?
http://localhost:4000/./callerand/authenticated-callerbeneath each new API prefix. Anonymous calls returnactor: nullfrom/caller;/authenticated-callerreturns a Problem Details error without an authenticated actor.Authorization: HASH-Service <secret>together withX-Authenticated-User-Actor-Id.📹 Demo
No recording attached. The local Scalar viewer at
/provides the interactive example.